I'm trying to write a shader that does 8-direction sprite billboarding such as in the original Doom. The obvious idea is to try to use a dot-product between the sprite forward-vector and the camera forward-vector, and use this to change the texture coords on my spritesheet. I just cant figure out this math…Even though I've done a lot of searches…

Matt_Aufderheide said:
use a dot-product between the sprite forward-vector and the camera forward-vector

That will tell you what angle to use, but not the left/right direction. You will need to use a 3rd vector, like the cameras cross product vector also against the sprites forward vector.

And you could have a bunch of if-then statements in your shader to determine which sprite to use. Such as

if (camForwardDotEnemyForward < -(cos(PI / 8))
//use forward facing sprite
else if (etc...)

I'm not sure if that is mathematically correct, even after editing it more than I care to admit.

I didn't have to use the “camSideDotEnemyForward” vector for the front facing sprite since it doesn't have a mirrored sprite.

I implemented this with acos. The result is the slice index [0,1,2,3….impostorCaptureAngles) :

float3 origin = mul(WORLD, float4(0, 0, 0, 1)).xyz;
float3 up = normalize(mul((float3x3)WORLD, float3(0, 1, 0)));
float3 face = mul((float3x3)WORLD, g_xCamera_CamPos - origin);
face.y = 0; // only rotate around Y axis!
face = normalize(face);
float3 right = normalize(cross(face, up));
pos = mul(pos, float3x3(right, up, face));
 // vertex position in local space -> world space, facing camera

// Decide which slice to show according to billboard facing direction:
float angle = acos(dot(face.xz, float2(0, 1))) / PI;
if (cross(face, float3(0, 0, 1)).y < 0)
{
	angle = 2 - angle;
}
angle *= 0.5f;
slice = floor(angle * impostorCaptureAngles);
 // impostorCaptureAngles = number of angles / texture slices

Thanks guys, I got it working using this method.