maybe it's this:

t_vec3 rotatedD0 = vec_mult(d.x, quadOrn[0]);

which should do float x vec3, but your code may assume vec3 x vec3?

And for this

vec_add(rotatedD0, rotatedD1);

assumed the second operand is added to the frist which is modified. Maybe should be:

rotatedD0 = vec_add(rotatedD0, rotatedD1)

I think the functions are correct.

t_vec3 vec_add(t_vec3 vec1, t_vec3 vec2)
{
	t_vec3 new_vec;

	new_vec.x = vec1.x + vec2.x;
	new_vec.y = vec1.y + vec2.y;
	new_vec.z = vec1.z + vec2.z;
	return (new_vec);
}


t_vec3 vec_mult(float m, t_vec3 vec)
{
	t_vec3 new_vec;

	new_vec.x = m * vec.x;
	new_vec.y = m * vec.y;
	new_vec.z = m * vec.z;
	return (new_vec);
}

But one of my assumption was wrong. Fix:

rotatedD0 = vec_add(rotatedD0, rotatedD1);

rotatedD0 = vec_add(rotatedD0, rotatedD2);

does it work now?

… also, if you give input of (0,0,1), rotating around Z will do nothing of course. : )

So lets rotate around x instead:

void		move_square(void *obj, int key)
{
	float a = 0.01f;
	
	/*t_vec3	rotZ[3] = {
		(cos(a),sin(a),0),
		(-sin(a),cos(a),0),
		(0,0,1)
	};*/
	
	t_vec3	rotX[3] = {
		(1,0,0)
		(0,cos(a),sin(a)),
		(0,-sin(a),cos(a)),
	};
	
	if (key == KEY_L)
	{
		t_vec3 d = (t_square*)obj)->vec;
		t_vec3 rotatedD0 = vec_mult(d.x, rotX[0]);
		t_vec3 rotatedD1 = vec_mult(d.y, rotX[1]);
		t_vec3 rotatedD2 = vec_mult(d.z, rotX[2]);
		rotatedD0 = vec_add(rotatedD0, rotatedD1);
		rotatedD0 = vec_add(rotatedD0, rotatedD2);
		((t_square*)obj)->vec = rotatedD0;
	}
}

does it work? We can extend this to rotate around all axis easily after that…

That's right, I didn't keep track of the additions. My square doesn't disappear. However, there is no visible rotation.
The vectors / axis before rotation:
vec(0,0,1)
right(-1,0,0)
up(0,1,0)

And after:
vec(0,-0.000698,-0.069753)
right(1,0,0)
up(0,1,0)

ellenature said:
However, there is no visible rotation.

See post above in case you missed it.

Okay, it works for the vector (1,0,0) with both rotX and rotZ but if the vector is (0,0,1) it works for neither.
Also, it rotates 90 degrees. What if you want a smaller angle?

ellenature said:
Okay, it works for the vector (1,0,0) with both rotX and rotZ but if the vector is (0,0,1) it works for neither.

Makes no sense… can't find a bug in the math.

Also, it rotates 90 degrees. What if you want a smaller angle?

No sense either, eventually ther is something wrong with your axis construction in tracing, but does not look like that.

It should rotate in small angles. Can you log some numbers of rotatedD0 within a series of function calls? It should gradually change.

Eventually also RotX vectors. Maybe your vec3 constructor does something unexpected. Yeah, i'd do this first.

ellenature said:
Also, it rotates 90 degrees.

This could be caused from tiny numbers, normalized here:

normal = get_normalised(square->vec);

So yeah, some logging should reveal it…

void move_square(void *obj, int key)
{
	float a = M_PI / 45;


	t_vec3 d = ((t_square*)obj)->vec;

	t_vec3 rotX[3] = {
		(t_vec3){1,0,0},
		(t_vec3){0,cos(a),sin(a)},
		(t_vec3){0,-sin(a),cos(a)}
	};
	
	t_vec3 rotY[3] = {
		(t_vec3){cos(a),0,-sin(a)},
		(t_vec3){0,1,0},
		(t_vec3){sin(a),0,cos(a)}
	};
	
	t_vec3 rotZ[3] = {
		(cos(a),sin(a),0),
		(-sin(a),cos(a),0),
		(0,0,1)
	};

	if (key == KEY_L)
	{
		t_vec3 rotatedD0 = vec_mult(d.x, rotY[0]);
		t_vec3 rotatedD1 = vec_mult(d.y, rotY[1]);
		t_vec3 rotatedD2 = vec_mult(d.z, rotY[2]);
		rotatedD0 = vec_add(rotatedD0, rotatedD1);
		rotatedD0 = vec_add(rotatedD0, rotatedD2);
		((t_square*)obj)->vec = rotatedD0;
	}
	else if (key == KEY_I)
	{
		t_vec3 rotatedD0 = vec_mult(d.x, rotX[0]);
		t_vec3 rotatedD1 = vec_mult(d.y, rotX[1]);
		t_vec3 rotatedD2 = vec_mult(d.z, rotX[2]);
		rotatedD0 = vec_add(rotatedD0, rotatedD1);
		rotatedD0 = vec_add(rotatedD0, rotatedD2);
		((t_square*)obj)->vec = rotatedD0;
	}
	else if (key == KEY_O)
	{
		t_vec3 rotatedD0 = vec_mult(d.x, rotZ[0]);
		t_vec3 rotatedD1 = vec_mult(d.y, rotZ[1]);
		t_vec3 rotatedD2 = vec_mult(d.z, rotZ[2]);
		rotatedD0 = vec_add(rotatedD0, rotatedD1);
		rotatedD0 = vec_add(rotatedD0, rotatedD2);
		((t_square*)obj)->vec = rotatedD0;
	}
}

It's working, the problem was the matrix syntax.
Why are the matrices different from these? : https://fr.wikipedia.org/wiki/Matrice_de_rotation#Les_matrices_de_base
How can we rotate in the opposite direction?
For a "diagonal" rotation (like in the video), I find the effect strange. Is it normal? If not, how can we fix it?