A while back, Crytek proposed storing vertex tangent / bitangent / normals in a quaternion. This seemed like a sensible approach, and my implementation worked well, most of the time. However, after using this with some more organic shapes, this seems like a totally unworkable approach.

I calculated the quaternion by creating a 3x3 matrix from the TBN vectors and retrieving the rotation of the matrix. A quaternion can only be used to describe the TBN vectors if they are orthogonal to one another. If the vectors aren't 90 degrees apart, the results will be wrong. I noticed because some organic shapes were displaying really bad normals.

If this is the case, why is anyone even talking about this approach? It seems useless. I suppose you could construct the 3x3 matrix in such a way that the error is all packed into the tangent and bitangent, so at least the normal is correct and all the error will be in the normal mapping.

Josh Klint said:
I suppose you could construct the 3x3 matrix in such a way that the error is all packed into the tangent and bitangent

Actually you could get the tangent right in any case too? Just the bitangent would need an additional number for an eventual sheer?
That's still 5 numbers vs. 9.

However, i guess they made sure their UV maps have minimized stretching, and so they could ignore the error in practice.

I don't think so, because there are many situations where the dot product of the tangent and the normal is not zero. However, if I use ensure the mat3 aligns to the normal and let the tangent and bitangent absorb the error, like I was saying, then it seems to look okay.

Josh Klint said:
I don't think so, because there are many situations where the dot product of the tangent and the normal is not zero.

Yes, but only if artists (or tools) violate the ideal of UVs with minimized distortion, eventually on purpose.

This often makes sense, and eventually some users of your engine might think of enforced quaternions as a limitation. I assume it's rarely a problem in practice, but i do not really have experience with artwork on normal maps and can't tell. Ideally you would make it optional to the users, i guess.

But i do think alignment to normal and tangent works technically.
Even if they are not orthonormal, the projection of the tangent to the plane of texture can still match its desired direction exactly. So we get only a difference in magnitude, but afaik this would not matter since we only care about direction and normalize anyway.
To get the bitangent right as well, you could use a given signed sheer value like this: bitang = normalize (tangent.cross(normal) + tangent * sheer).

But maybe there are some special cases where magnitude matters, like anisotropic hair shaders or such stuff? Not entirely sure about that.

Edit: Maybe i think too much about texture space, ignoring the problem of mapping from that back to object space.
Related issues could be solved with compensating the normals in the texture map. But if those normals are available only in compressed formats, such preprocessing step would certainly cause a quality loss.