Lets say we have a global coord system and local coord system and that local coord system is rotated by some amount by some arbitrary vector.
How could I find this rotation vector and angle given the two coord systems ?

The amount of rotation does not depend on the coordinate system. The vector around which you are rotating should be expressed in the basis of the appropriate coordinate system. You normally convert between coordinate systems by a matrix multiplication, but I am not entirely sure what format you have your coordinate systems in.

Matrix solution would be ok, but I was looking for a quaternion solution.

Ill try to give an eg. Lets say we have a normal coord system, and a vector defined in it as v(-0.5, 0, -1). And that there is another coord system (of an object) located at (2, 1, 1) and it is rotated by facing directly on opposite z, that is its x and y are the same as original coord system but its z is pointing in (0, 0, -1) in respect to original coord system.

What I want to know what are the coords of vector v in respect to the second coord system.

The location of the origin of the coordinate systems is irrelevant for this particular problem, because we are converting only vectors. When you say "and it is rotated by facing directly on opposite z", you have to realize that that transformation is not a rotation at all: It's a reflection. The coordinates of v in the second coordinate system are (-0.5, 0, 1). If v is the axis of a rotation, you'll actually have to negate the rotation amount because the coordinate systems have different orientation.

Hmm, Im all confused.

So you say the coords of vector v remain unchanged. I don't understand this.

Lets then say that a second coord system is not a reflection but a rotation with its forward vector (z component) its facing (0.5, 0, -1), and its up (y) and right (x) are orthonormalized in respect to forward (z), that is they are the same as in first coord system. What would than be the coord of a vector v ?

Would it still remain the same ?

I don't mean the rotation around vector v, just its coords in respect to second coord system.

So you say the coords of vector v remain unchanged.

No, that's not what I say. The sign of z changed.

Lets then say that a second coord system is not a reflection but a rotation with its forward vector (z component) its facing (0.5, 0, -1), and its up (y) and right (x) are orthonormalized in respect to forward (z), that is they are the same as in first coord system.

I don't understand that description of a coordinate system. Just tell me what vector is (1,0,0) in the new coordinates, what vector is (0,1,0) and what vector is (0,0,1). From that we can build a 3x3 matrix and do all the operations easily.

So you say the coords of vector v remain unchanged.

No, that's not what I say. The sign of z changed.

Lets then say that a second coord system is not a reflection but a rotation with its forward vector (z component) its facing (0.5, 0, -1), and its up (y) and right (x) are orthonormalized in respect to forward (z), that is they are the same as in first coord system.

I don't understand that description of a coordinate system. Just tell me what vector is (1,0,0) in the new coordinates, what vector is (0,1,0) and what vector is (0,0,1). From that we can build a 3x3 matrix and do all the operations easily.

Oh ok. To a response on your first quote.

I didnt notice that sign change, things are starting to be a little more clear now. So my question now is how did you do that ? Is there some matrix, quaternion inversion or maybe something else ?

On a second quote.

I meant for second coord system to be some object in space with its forward (z), up (y) and right (x) as a coord system. So for this object in its local coord system its basis are forward (0, 0, 1), up (0, 1, 0), right (1, 0, 0). But in terms of first coord system (the one where this object is placed) its basis are forward (0.5, 0, -1), up (0, 1, 0), right (1, 0, 0).

This time in second quote the object is not translated, its located at origin (for easier rapresentation). Note that its up and right are unchanged from first coord system, also for easier rapresentation.

So the thing I was actualy looking for is the vector v in terms of this object, that is its local coord system.

You can just write down the matrix if you know what it does to (1, 0, 0), (0, 1, 0) and (0, 0, 1), it is just the images of those vectors after the transform as the rows of the matrix.

So to map (1, 0, 0) to (1, 0, 0); (0, 1, 0) to (0, 1, 0) and (0, 0, 1) to (0.5, 0, -1) you would use the matrix

(1, 0, 0)

(0, 1, 0)

(0.5, 0, -1)

and to map back you use the inverse of that matrix.

The matrix has determinant -1 (see http://www.wolframalpha.com/input/?i=det{{1%2C+0%2C+0}%2C+{0%2C+1%2C+0}%2C+{0.5%2C+0%2C+-1}} ), so it corresponds to a reflection and the orientation is flipped. You can't represent that transform with a unit quaternion since unit quaternions correspond to rotations and they all have determinant +1.

The problem with the matrix in Paradigm Shifter's post is that it is not orthonormal. If you want your rotations to still be recognizable as rotations in the new coordinates, you need to use an orthonormal basis.

Yeah I was gonna mention that at the end but deleted it.

You can make an orthonormal basis from independent ones using cross products and normalisation or use the Gram-Schmidt process http://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process

Unit quaternion rotations always give an orthonormal matrix (ignoring floating point inaccuracy) but you can't represent a reflection unless you also use a scale matrix to scale with a negative determinant.