Hi,

I'm doing a thing where I simulate looking at 3d objects from different perspectives using billboards and sprite sheets like your old Doom or MarioKart.

the way it works is in the shader, I derive a latitude / longitude (or pitch/yaw) between the camera and the billboard, and map that to an offset in the spritesheet.

this is working to my satisfaction, except that right now it is limited in a way that the objects need to be upright. the camera also can't roll. If the object has rotation around anything but the y-axis, or if the camera up-vector isn't [0,1,0] the rotation becomes borked. Of course, I want to support any rotation about any axis and full camera freedom.

It's easy to see that conceptually, all I have to do is get 3 angles instead of 2, roll, pitch and yaw, and then rotate the billboard sheet (or the uv lookup) by the roll angle, and use pitch and yaw for texture lookup as before.

The trick is, and this is where my brain stops functioning, how do I get these 3 angles that depend on each other?

I'm thinking the object's translation / rotation has to be transformed into view space (or projection space? not sure)

basically the roll angle is going to be a rotation around the vector from the eye to the object, as that will just rotate the 2d sprite. and the pitch / yaw will somehow have to take that view direction into account as well. if the roll angle changes, the yaw/pitch will have to be influenced correctly by that.

hopefully, to you, dear reader, this is a trivial or at least solvable problem and you can help me out. 👍

Seems a problem of euler angles, solvable with trial and error.

I have some tools which might help:

__forceinline void FromEuler (const sVec3 &radians, const int order = 0x012) 
	{
		int a[3] = {(order>>8)&3, 
					(order>>4)&3, 
					(order>>0)&3};

		sMat3 r[3] = {	rotationX (radians[0]),
						rotationY (radians[1]),
						rotationZ (radians[2])};

		(*this) = r[a[0]];
		(*this) *= r[a[1]];
		(*this) *= r[a[2]];
	}

	__forceinline sVec3 ToEuler (const int order = 0x012) 
	{
		int a0 = (order>>8)&3;
		int a1 = (order>>4)&3;
		int a2 = (order>>0)&3;

		sVec3 euler;
		float d = (*this)[a0][a2];
		// Assuming the angles are in radians.
		if (d > (1.0f-FP_EPSILON)) 
		{ // singularity at north pole
			euler[a0] = -atan2f((*this)[a2][a1], (*this)[a1][a1]);
			euler[a1] = -3.1415926535897932384626433832795f/2.0f;
			euler[a2] = 0;
			return euler;
		}
		if (d < -(1.0f-FP_EPSILON))
		{ // singularity at south pole
			euler[a0] = -atan2f((*this)[a2][a1], (*this)[a1][a1]);
			euler[a1] = 3.1415926535897932384626433832795f/2.0f;
			euler[a2] = 0;
			return euler;
		}
		euler[a0] =	-atan2f(-(*this)[a1][a2], (*this)[a2][a2]);
		euler[a1] =	-asinf ( (*this)[a0][a2]);
		euler[a2] =	-atan2f(-(*this)[a0][a1], (*this)[a0][a0]);
		return euler;
	}

Those are member functions of a 3x3 matrix calss for orientation and using a vec3 for euler angles.

the order parameter allows easy trial and error. 0x012 means XYZ order, 0x210 means ZYX order and so on. So you usually have 6 options to try.

However, the code can not handle orders like XYX, with one axis appearing twice. But i assume yours is not such a special case.

I guess you might end up with something like: Calc rotation matrix from camera space to object space, get euler angles from this rotation using a certain order, use angles x,y to index the texture and angle z to rotate it.

So maybe the code helps to make something work and then you could optimize for your case from there.

Have you managed to make any progress on this mutable-copy?
At first glance your concept is sound, you just need to be able to go behind (in world space) a sprite and add roll.

I'm curious about the billboard, is this camera facing?
Is the 3d Box shown just a bounding box? or is the billboard aligned to it?

Based on your second image, it looks like you've got the pitch and yaw working fine, maybe ignore roll until you're happy with the basic 2D navigation of your sprite. As you've said, the roll should be a simple rotation of the sprite sheet. You could rotate the billboard or possibly better still just rotate the camera?
The only issue I can spot may come from the effect the cameras FOV has on the billboard.

right now it is limited in a way that the objects need to be upright.

Why can't you flip the sprite sheet upside down when you reach the UV pitch extremes? If you flip it and bounce the direction of your UV pitch look up, this issue should be solved (this applies to your displayed sprite, i can explain further if you need).
Albeit this will only work with a sprite that is mirrored (front to back), or you need a full 360 sprite sheet.