I'm currently deriving the lookat quaternion by computing the 3x3 orthonormal camera matrix and then extracting the quaternion from it.
I'm interesed in computing the lookat quaternion directly though. I wrote this function (without optimizations and ignoring edge cases):

// Camera at point 'camera' looking at the point 'target', the "sense" / "hint" vector is 'up' (e.g. no roll for vec3(0.0, 1.0, 0.0)).
quat LookAt(const vec3& camera, const vec3& target, const vec3& up)
{
    // For simplicity, ignoring edge case where worldF and cameraF (or up) are colinear
    vec3 worldF = vec3(0.0, 0.0, 1.0);
    vec3 cameraF = - normalize(target - camera);
    float cosAngle = dot(worldF, cameraF)
    vec3 axis = normalize(cross(worldF, cameraF));
    quat fixRoll = quat::identity; // ???
    return fixRoll * quat::FromAxisAngle(axis, acosf(cosAngle)); // maybe right mult?
}

My question is: what should be the quaternion fixRoll that aligns the y camera's vector with the given 'up' vector?

I tried to rotating around worldF and cameraF axis in few flavors.
I tried to multipling fixRoll from left (as showed above) and from the right.

Would be wonderful if someone can show how to calculate the quaternion that corrects the camera roll and aligns the local y vector with the given 'up' and provide an explanation.
Thanks!
 

First of all, there is (IIRC, and I think I do) a nice algorithm for directly computing a quaternion that rotates from one vector to another, without any trigonometry. Perhaps among other things, it has the advantage of having fewer special cases than the axis-angle method.

As for correcting the roll, my first thought is that you'd want to apply a rotation that rotates the up vector associated with the first quaternion onto the desired up vector (which would be the same up vector you'd compute in a matrix or vector version of the algorithm). The same 'rotate between vectors' algorithm can be used to compute this quaternion.

To cover myself here, let me stress that that's just off the top of my head, and I may have missed something. Perhaps there's a better solution.

In any case, when I looked into this myself in the past, I think my conclusion was that computing the quaternion directly didn't really offer any advantages (in terms of simplicity, number of operations, number of special cases, etc.) over extracting the quaternion from a matrix. Quaternions are often (perhaps usually) converted back to matrix form eventually, and going from matrix to quaternion to matrix is a bit roundabout, but since quaternions can be a useful intermediate form, I don't think there's anything wrong with doing it that way.

I just searched online for 'quaternion look at' and looked at a few threads, and I didn't see anything that was obviously inconsistent with what I've said here. Again though, this is just off the top of my head, and I may very well be missing something.

Quote

First of all, there is (IIRC, and I think I do) a nice algorithm for directly computing a quaternion that rotates from one vector to another, without any trigonometry. 

It is called 'Shortest Arc Quaternion'. E.g. Stan Melax wrote about it in one of the Game Programming Gems (e.g. GPG 8 I think).

4 minutes ago, Dirk Gregorius said:

It is called 'Shortest Arc Quaternion'. E.g. Stan Melax wrote about it in one of the Game Programming Gems (e.g. GPG 8 I think).

It may be in GPG 8, but it's also in GPG 1, for what it's worth (just in case the OP goes looking for it).

Thank you for the responses.
The shortest arc quaternion also produces a roll unfortunately, so we're back to square one.

@Zakwayda, it's really interesting to hear that calculating the matrix and extracting the quaternion was actually more efficient than calculating it directly.. I'm going to profile it once the function is completed.

The goal of the question is to specifically calculate the "fixing" quaternion.

Quote

The shortest arc quaternion also produces a roll unfortunately, so we're back to square one.

I think there was probably a miscommunication here. The 'shortest arc' algorithm is simply a possible (perhaps preferable) replacement for how you're currently building your component quaternion(s). It doesn't have any direct bearing on the roll issue.

Regarding comparing the various methods for performance, operation count, and so on, I'm just going off my recollection and my intuitive sense of what's involved in both methods, so I would certainly double-check that independently (as you said you intend to do). I'd also suggest that, depending on the circumstances, with respect to this operation, fine-grained efficiency may not matter much unless you're performing the operation many times per update (e.g. hundreds or thousands).

Quote

The goal of the question is to specifically calculate the "fixing" quaternion.

For what it's worth, I did address that in my previous post. Also, my search for 'look at quaternion' led me to some other resources on the same issue, so you might dig around a bit online and see what you can find.