Solved: didn't think clearly and realized I can't just compare the cross-product with 0,0,0.  

Fixed by doing this:

float3 originVector = float3(0.0, 0.0, 0.0) - v1.xyz;

if (dot(cross(e1, e2).xyz, originVector) > 0.0)
{
   //...
}

I'm trying to write a geometry shader that does backface culling. (Dont ask me why)

What I'm doing is checking the cross-product of two edges of the triangle (in NDC space) and checking if it's facing 0,0,0 .
The problem is when I compile I get this error:

Quote

...\GeometryShader1.hlsl(12,36-41): error X4577: Not all elements of SV_Position were written

this is i guess because if it isn't facing us, I dont append any verts to the stream. I always assumed maxvertexcount implied I can emit as few verts as I like, but I suppose not.

How do I get around this?

Shader below:

struct GS_IN_OUT
{
	float4 Pos : SV_POSITION;
	float4 PosW : POSITION;
	float4 NorW : NORMAL;
	float2 UV : TEXCOORD;
};

[maxvertexcount(3)]
void GS_main(
	triangle GS_IN_OUT input[3],
	inout TriangleStream< GS_IN_OUT > output
)
{
	//Check for backface
	float4 v1, v2, v3;
	v1 = input[0].Pos;
	v2 = input[1].Pos;
	v3 = input[2].Pos;

	float4 e1, e2;
	e1 = v1 - v2;
	e2 = v1 - v3;

	if (dot(cross(e1, e2).xyz, float3(0.0, 0.0, 0.0)) > 0.0)
	{
		//face is facing us, let triangle through
		for (uint i = 0; i < 3; i++)
		{
			GS_IN_OUT element;
			element = input[i];
			output.Append(element);
		}

	}
}

I started looking at it. Geometry shader are usually the worst in every scenario with GCN hardware but behave better for culling than geometry expansion so what you want to do should work.

I get the same error message as you ( also you have a bad cross of float4 and truncation ).

I will dig to it tomorrow once back home

19 hours ago, galop1n said:

I started looking at it. Geometry shader are usually the worst in every scenario with GCN hardware but behave better for culling than geometry expansion so what you want to do should work.

 

I get the same error message as you ( also you have a bad cross of float4 and truncation ).

 

I will dig to it tomorrow once back home

Don't know if you missed it but I solved it by replacing the 0,0,0 vector in the dot()-evaluation with an actual vector (0,0,0 - vertexpos).
I assume the reason it gave me the error is that it detected that all verts would fail that test.

Also I dont see any problems with the cross, both arguments are float4 and then I explicitly get the xyz from it.

Oh yeah the dot !

cross in hlsl is only defined for float3, so your float4 are implicitly truncated, and .xyz on output is a no op.

Also float4(0,0,0,0) is not necessary in hlsl because glsl sucks. You can write 0 and it self expand