I currently attempt to rotate a object in the game to face toward another object, but stuck on

how to find angle on x axis between two vectors(if that makes sense).

The final transform for the object to be rotated( and moved) is

translatation matrix * rotation matrix on y axis(0,1,0) * rotation matrix on x axis(1,0,0)

So it can be rotated left and right, and up and down to face its target when moving to it. I figured out in order to rotate on y axis, I need to make the y component of the object's target direction vector ( target position - object position ) zero, like vec3(targetDirection.x, 0, targetDirection.z), and find the angle between target direction vector and the object's default direction vector, which is always (0,0,-1). It works fine but I can't figure out how to find the angle on the x axis. Using the same trick, making the x component zero instead of y component, it doesn't work.

The two vectors are target direction vector ( target position - object position) and object's default

direction vector which I always set to vec3(0,0,-1). Thanks.

From what you're describing it sounds like you're basically implementing a lookAt.

Take a look toward the bottom of this page to see how such a matrix may be constructed:

https://msdn.microsoft.com/en-us/library/windows/desktop/bb204936%28v=vs.85%29.aspx

Edit:

The chirality of this was bothering me, so I made a test to check it out. It's left-handed, but invert the "eye" and "at" vectors when calculating the new z axis vector. This was the function I ended up with (in Unity, but the math is the relevant part):

    Matrix4x4 makeMatrix(Vector3 position, Vector3 target)
    {
        Vector3 z = (position - target).normalized;
        Vector3 x = Vector3.Cross(Vector3.up, z).normalized;
        Vector3 y = Vector3.Cross(z, x);

        Matrix4x4 mat = Matrix4x4.identity;
        mat[0, 0] = x.x;    mat[0, 1] = y.x;    mat[0, 2] = z.x;
        mat[1, 0] = x.y;    mat[1, 1] = y.y;    mat[1, 2] = z.y;
        mat[2, 0] = x.z;    mat[2, 1] = y.z;    mat[2, 2] = z.z;
        mat[3, 0] = -Vector3.Dot(x, position);
        mat[3, 1] = -Vector3.Dot(y, position);
        mat[3, 2] = -Vector3.Dot(z, position);

        return mat;
    }

To find axis / angle between 2 unit vectors:

vec axis = (from.Cross(to)).Unit(); // axis from normalised cross product
float angle = acos(from.Dot(to));

To find the angle on ANY other axis:

vec rotationVector = axis * angle;
float otherAngle = otherUnitAxis.Dot(rotationVector);

In your case (needing the x angle, so otherUnitAxis is (1,0,0)) you could simly use:

float xAngle = rotationVector.x; // or axis.x * angle


Let's make a function out of this

float AngleOverAxis (vec &from, vec &to, vec &axis)
{
float angle = acos(from.Dot(to));
vec rv = (from.Cross(to)).Unit() * angle; // todo: zero cross?
return axis.Dot(rv);
}

So, for your example you would need to first find the y angle:

xform = Identity; xform.pos = objectWorldPosition;
vec to = (targetPos - xform.pos).Unit(); // target dir
vec from = objectLookDir; // look dir
float yAngle = AngleOverAxis (from, to, vec(0,1,0));

Then rebuild the transform with this new y rotation to get an updated looking direction for the object:

matrix rotationFromYAngle = matrix::RotationY(yAngle);
xform *= rotationFromYAngle;
from = xform.Rotate(objectLookDir);

Finally find x angle with new looking direction.

float xAngle = AngleOverAxis (from, to, xform.Rotate(vec(1,0,0))); // <- bugfix here
matrix rotationFromXAngle = matrix::RotationX(xAngle);
xform *= rotationFromXAngle;

EDIT: oops, x vector must be rotated from local to global space

From what you're describing it sounds like you're basically implementing a right-handed (I think?) lookAt.

The look might not do it, because the OP seems to want 2 rotations in strict y, x order.
Look at would result in a single rotation, so it would be necessary extract y/x with a matrix to euler angles conversation with correct order.
(might be available in DX too)

From what you're describing it sounds like you're basically implementing a right-handed (I think?) lookAt.

The look might not do it, because the OP seems to want 2 rotations in strict y, x order.
Look at would result in a single rotation, so it would be necessary extract y/x with a matrix to euler angles conversation with correct order.
(might be available in DX too)

I didn't interpret his post that way. He began with

I currently attempt to rotate a object in the game to face toward another object...

and then went on to explain the method he was attempting to use.

I cover this in my Game Dev Math Recipes, specifically Rotating to face another object. All "recipes" are explained with a full JavaScript running example.