AntiTwister said:

There is a nice set of 24 unit quaternions which are spaced apart evenly, called the Hurwitz quaternions. This is a handy choice to use for a base bunch of orientations to render your object at.

If you have the quaternion representing the orientation at runtime, it is then easy to take a dot product between that orientation and all the orientations you have pictures for. Whichever dot product is the largest is the best pre-rendered picture to use!

Huh, that actually sounds interesting enough that I might end up looking into it. Thanks for the reference to the Hurwitz quaternions - it gives me a term to search for instead of trying to bruteforce search terms for what I'm doing haha.

The thing about finding the largest dot product sounded a bit brutal at first, but then I remembered that my atan2 implementation requires a division operation, which is quite slow on my target platform, so it might work out anyway.

JoeJ said:

I can't imagine what your problem might be, but hacking the function probably isn't needed. […]

Here is what the scene looks like, with the navmesh and camera:

(just pretend that the yz axes are swapped; I'm accounting for this during export)

Notice how the camera is pointed towards -z, which is what I mentioned is causing issues for this particular case (all my other tests went fine, with the camera pointed towards +z).

Now, the resulting matrix I get (after a bit of further manipulation) is:

-0.823, 0.25, -0.511
-0.07, -0.936, -0.347
0.564, 0.249, -0.787

Quick notes: The y component (second row) is negated because of some implementation details (I need y pointing down the pre-rendered image), but that's not too important. And the z component (third row) is pointing into the screen, rather than out of the screen as is standard, because that makes the most sense for me (again, not too important).

Anyway: Based off your function implementation, the z rotation angle (in x→y→z rotation order) would be -atan2(-(0.25), -0.823) = 163.1 degrees, which is very obviously wrong by comparing against the image above. If I instead use -atan2(-0.25, 0.823), though, I get 16.9 degrees, which /is/ correct. This is what I meant about the atan2 x argument being “contaminated” when the camera points in the -z direction - the y-axis rotation needed to point that way will negate the M[0][0] component (which makes sense, since you're now looking at things “backwards”).

Though now that I think about it a bit more… shouldn't it be possible to correct for his, by just inspecting the y rotation angle? ie. if cos(r_y) < 0, then r_z = pi - r_z? (I'd check the angle directly rather than evaluating the cosine, but just to make it clearer)

Aikku said:
Though now that I think about it a bit more… shouldn't it be possible to correct for his, by just inspecting the y rotation angle? ie. if cos(r_y) < 0, then r_z = pi - r_z? (I'd check the angle directly rather than evaluating the cosine, but just to make it clearer)

Hmm, your report still doesn't ring any bells on me. Never had the problem that just one angle is wrong and should be 180-x.

But my best guess is that it's about left / right handed coordinate system convention being wrong.
This can be fixed by negating one basis vector of the matrix before using the function with it.
I guess it also works no matter of you negate one column or one row, but again there is trial and error to figure out which one.

JoeJ said:

But my best guess is that it's about left / right handed coordinate system convention being wrong.

I'd thought that for a bit… but like I said, your suggested implementation does work for all my test cases, /except/ for the one where the camera is pointing backwards. If I negate any of the terms in the atan2() or swap x/y around, then /everything/ breaks, so I'm leaning towards it being correct (especially since the z rotation involves only the first row of the matrix, which both left-handed and right-handed systems agree on, even disregarding my flipped y axis).

JoeJ said:

but again there is trial and error to figure out which one.

Haha, yeah, I'm definitely feeling that one.

I'm thinking that if you're extracting these angles and then re-combining them (possibly after some manipulation), it all just works itself out because the angles were all calculated from the same “basis” (I don't think that's the right word, but it felt the most accurate; ie. they all “agree with each other”), so combining all of them back together “just works”. But if you need the angles in isolation rather than all working together, there might be some need for correction.

But anyway: I guess as a quick and dirty hack, I could just take the absolute value of cos(r_z), since I don't expect to be flipping the camera upside-down. Thanks for the help!