Hello.

I learned how to make a rotation matrix from a normal, but I still don't understand how to control rotation around its own axis. I'm trying to fix the tanget, but nothing works for me. I tried in two ways:

1. I take the tangent as the cross product of up and the normal. I make a separate Y rotation matrix by the angle between the tangent and the X axis in the XZ plane. I multiply the TBN matrix by it. As a result, there is an unnecessary full 360 rotation around its axis with the normal opposite to Up:

Code:

void  physics::rotationMatrixFromVectorY_UP_21(glm::vec3& n, glm::mat3& mat) {

glm::vec3 normal = glm::normalize(n);
glm::vec3 up = glm::vec3(0.0f, 1.0f, 0.0f);

glm::vec3 tangent = glm::normalize(glm::cross(up, normal));
glm::vec3 binormal = glm::cross(normal, tangent);

mat[0][0] = tangent.x;
mat[0][1] = tangent.y;
mat[0][2] = tangent.z;
mat[2][0] = -binormal.x;
mat[2][1] = -binormal.y;
mat[2][2] = -binormal.z;
mat[1][0] = normal.x;
mat[1][1] = normal.y;
mat[1][2] = normal.z;

float angleDir = std::atan2(tangent.x, tangent.z);
angleDir += glm::radians(DEBUG_IMGUI_UGOL_2);
glm::mat3 mmmm = glm::mat3(glm::rotate(glm::mat4(1.0f), -angleDir, glm::vec3(0, 1.0f, 0)));
mat *= mmmm;
}

2. I take the tangent as normalize (normal.y, -normal.x, 0.0f). To be in the YX plane all the time. Result:

Code:

void  physics::rotationMatrixFromVectorY_UP_20(glm::vec3& n, glm::mat3& mat) {

glm::vec3 normal = glm::normalize(n);
glm::vec3 tangent = glm::normalize(glm::vec3(normal.y, -normal.x, 0.0f));
glm::vec3 binormal = glm::cross(normal, tangent);

mat[0][0] = tangent.x;
mat[0][1] = tangent.y;
mat[0][2] = tangent.z;
mat[2][0] = -binormal.x;
mat[2][1] = -binormal.y;
mat[2][2] = -binormal.z;
mat[1][0] = normal.x;
mat[1][1] = normal.y;
mat[1][2] = normal.z;
}

As you can see in the video, I can't constantly fix the tangent so that the ax does not rotate around its axis (Y). I need to complete my code, but I don't understand how. I feel like I should be using dot product, but my knowledge of mathematics is lacking. Please help me.

It's not possible to prevent flipping if you construct a tangent for given normal, but if your tangent is free, you can minimize it's change over time with a changing normal. To do this, you need to store the tangent (or the whole matrix), and keep rotating it each frame so one chosen axis matches given normal. If this rotation has the minimum possible angle, the tangent movement is minimized as well. Thus you calculate the axis and angle rotation from previous to current normal.

vec3 prevNormal = myMatrix[0]; // we use x axis for the normal, and store myMatrix somewhere.
vec curNormal = ...; // given

vec3 axis = normalize(cross(prevNormal, curNormal));
float angle = acos(dot(prevNormal, curNormal));
quaternion rot = quaternion::angleAxis(angle, axis);
matrix3 m (rot);

myMatrix *= m;

Code may be buggy, e.g. wrong orders in cross product or matrix multiply, but you get the idea.
It's also possible to construct rotation matrix directly without a need for quaternions.

After that your tangent should move as expected, with no flipping. But you have no control over the tangent, which can be a problem too. It always depends on your application.