Hi, I need help with understanding what is wrong with my hlsl shader (ps_3_0 for d3d9).

Here is part of my shader.
It makes color part middle of the quad in GREEN.
Make outside of sides of the “ships” RED.
Make space between ships BLUE.
As you see on one of screenshot.

float4 PS_CivilianTrafficConnectionLine_Corris(VS_OUT In) : COLOR
{
float blindZoneStart = CTCL_SolidZone + CTCL_HalfZone;
float blindZoneEnd = 1.0f - blindZoneStart;
if (In.Tex.x > blindZoneStart && In.Tex.x < blindZoneEnd)
{
return float4(0.0f, 1.0f, 0.0f, 1.0f); // WIDE GREEN LINE (AT BLINDZONE)
}
else
{
// ...
if (shipUnderPixelOnLine.shipIndex == -1)
{
return float4(0.0f, 0.0f, 1.0f, 1.0f); // BLUE SPACE BETWEEN SHIPS (swap with discard)
// BUGGED DISCARD - TURN ON TO SEE THE BUG
// discard;
// return float4(0.0f, 0.0f, 1.0f, 1.0f);
}
else
{
// ...
if (In.Tex.y < (yLowHighBounds.x + marginY) || In.Tex.y > (yLowHighBounds.y - marginY))
{
return float4(1.0f, 0.0f, 0.0f, 1.0f); // RED SPACE OUTSIDE OF SHIP BoundBox
}
else
{
float4 color;
// ...
return color; // SHIP
}
}
}
}

technique CivilianTrafficConnectionLine
{
pass P0
{
ALPHATESTENABLE = TRUE;
ALPHABLENDENABLE = TRUE;
CULLMODE = NONE;
VertexShader = compile vs_3_0 VS_DrawOnScreen();
PixelShader = compile ps_3_0 PS_CivilianTrafficConnectionLine_Corris();
}
}

Problem is - If i change
return float4(0.0f, 0.0f, 1.0f, 1.0f);
to
discard;
return float4(0.0f, 0.0f, 1.0f, 1.0f);

I will see the bug. Instead of loosing all BLUE I loose most of GREEN as well.
And I don’t see the way it happened ?
Green part is in another part of the IF operator.

Result of the bug you may see on another screenshot.

Does anybody knows how this happening?

Shader compile with string:
fxc.exe /T fx_2_0 /Od /Fo Shaders.fxo /Zi /Fc Shaders.asm Shaders.vsh

I use PIX for windows to see what happening with rest of GREEN line - but there is nothing.
As if no shaders apply for pixels outside of small green line.
I think - because of ‘discard’.

I hope I can find here someone who remember ancient techs ).

discard and return should NOT be used together as they have different implication.
discard tells the fragment shader to ‘discard’ the result of shader invocation for current fragment. This means that fragment shader will not/should not write any output to the attached render target. Technically upon reaching a discard, the shader could end its execution and return immediately. The side-effect of this is that any statement after the discard will NOT be executed, which, if I understand correctly is what you're seeing. ‘return' is return a result. You cannot return and discard and the same time. Pretty much asking the shader to do 2 contradicting operation. However, seeing that the compiler didn't flag this , you would have to read into the HLSL documentation to see what the behavior in this case should be.

Not sure about discard directly, haven't used it personally. For not writing certain pixels based on a condition, I usually use “clip”.

PIXEL_OUT mainPS(GEOMETRY_OUT In)
{
	PIXEL_OUT Out;

	float4 vSampleResult0 = DefaultTexture0.Sample( Sampler0,  In.vSpriteTex);
	float4 vLerpToneResult0 = lerpTone(vSampleResult0, In.vColor);
	
	clip((vSampleResult0.w <= 0.5f) ? -1 : 1);
	Out.vColor.rgb = vLerpToneResult0;
	Out.vColor.a = 1.0f;
	
	return Out;
}

This discards all pixels whose alpha-value is below 0.5f. Try if clip fixes your issue, if it does, it's something to do with how discard is used. If not, something else is the problem.

@cgrant

Thanks for the answer.

Code like this:

discard;
return float4(0.0f, 0.0f, 1.0f, 1.0f);
Used because fxc compiler didn't compile shader without ‘return’ in each ‘if’ branch. (Compile error).
So this 'return' is useless and not executed, but needed to compile shader.

And if you look close code for GREEN part does not have ‘discard’ and it is in function begin.
So, GREEN part should not be changed compared to ‘all return’ version of the shader on first screenshot.
But - as you can see on second screenshot - GREEN become much shorter.

@Juliean

Thanx for an advise. Will try clip.

@Juliean You were right!

I changed all discard calls with clip() - everything is working now.

Thanks for a help.

@Corris Thanks for the insight. I was gonna hint at some form of compiler bug..