I think I've got it, thank you very much, it just needs to be multiplied with inverse matrix M.

But I think I set up my matrix wrong so Ill set it up again.

Original matrix M, we set its columns as right, up and forward:

(0.44, 0, 0.89)

(0, 1, 0)

(-0.89, 0, 0.44)

Inverse of M, M(-1):

(0.44, 0, -0.89)

(0, 1, 0)

(0.89, 0, 0.44)

So the result of multiplying M(-1)*v = v', where v' (0.67, 0, -0.88) and its the same vector in space.

Phew, that was hard work ;)

Glad you got it in the end though.

Are you learning linear algebra or just looking at game programming material? A bit of linear algebra would help you (especially so your terminology is understandable to the likes of Alvaro and myself). Note how we managed to say what you were trying to achieve in a few equations which are a lot less ambiguous than wordy descriptions.

EDIT: If you want the quaternion solution, it's fairly similar.

If you have

v_out = q * v_in * q^-1

then (multiplying on left by q^-1)

q^-1 * v_out = q^-1 * q * v_in * q^-1 = v_in * q^-1

(multiply on right by q)

q^-1 * v_out * q = v_in * q^-1 * q = v_in

so

v_in = q^-1 * v_out * q

an since quaternions are normally unit length then q^-1 = conjugate(q).

Note, this is similar to the matrix version because if you have

v_out = M * v_in then v_in = M^-1 * v_out. In the quaternion version, you have

v_out = q₁ * v_in * q₁^-1

and

v_in = q₁^-1 * v_out * q₁

then writing q₂ = q₁^-1 we have

v_in = q₂ * v_out * q₂^-1

which is the same equation as for v_out but with q₁ replaced with q₁^-1

EDIT2: The algebra works the same way as for matrices, you have to be careful whether you are multiplying on the right or the left hand side of each equation though, since matrices and quaternions aren't commutative (a*b != b*a in general) but are associative (i.e. you don't need brackets for multiplication since a*(b*c) = (a*b)*c, so you can write a*b*c with no ambiguity). A matrix and its inverse always commutes though, as does a quaternion and its inverse.

Well I must say thank you guys once again.

It feels so clear now and easy. It was some time since I learned that in college. I got pretty good grades too, probably I knew that stuff but when I had to implement it I stuck.

Quaternion math is pretty similar to matrix, although I did not understand all that you have written by looking at it. I did calculate some inverse quaternion and I got the same result.