Okay, so I officially got confused (which is most likely also due to actually working on something at around 5am).

So, my objects in scene (I call them "entities") stored in a scene-graph like way have a transformation matrix (aka world matrix - W). Now, whenever I pick some object, I want to allow to do the translation/rotation/scale either in default basis (Euclidean space axes), or local basis (basically Euclidean space axes transformed with rotational part of world matrix - let's call it L') ... e.g. not unlike 'Global' and 'Local' space in Unity.

I also know the transformation matrix - T - with which I want to transform the object. So the only remaining question is, how to calculate new world matrix for global and local - W'?

One of them is going to be W' = T * W (which I believe should be the local one) ... and the global one should therefore be W' = (inverse(L') * T) * W

Am I approaching this properly?

It is all about bringing the object of interest temporarily into a co-ordinate system where the applied transform has the desired effect. Let's say that the object of interest has currently a world matrix W, that the desired transform is T and should be applied in a reference system R.

Then the new world matrix is computed as (shown for column vector convention):  W' := W * R * T * inv( R ) * inv( W )

The part to the right of T undoes the current world transform with inv(W) and makes the reference system active with inv(R). The part to the left of T redoes that setup; therefore I said that it is only "temporarily".

Now the secret lies in how to choose R for the particular purposes. However, I have not really understood what exactly your L is.

Thanks for the response - so I can second that it definitely is viable to get some sleep first and think about it the next day (prevents huge amounts of confusion). I've written it down on the paper and it makes a lot more sense now.

Given a world matrix W:

1  0  0  0
0  0 -1  0
0  1  0  0
0  0  0  1

and transformation matrix M:

1  0  0  0
0  1  0  5
0  0  1  0
0  0  0  1

a result of W * M would be:

1  0  0  0
0  0 -1  0
0  1  0  5
0  0  0  1

Which is correct - the resulting matrix is M applied to transformation W in local coordinates. Now if I want different basis R - I need to transform M into it and then back from it like - R * M * inv(R) - which is pretty much equal to what you wrote (although I don't want to undo W transformation) - so, when I want global coordinates, my basis relatively to W is inv(W) - therefore W' = W * inv(W) * M * inv(inv(W)) = W * inv(W) * M * W = Id * M * W = M * W

To proof this:

M * W would be:

1  0  0  0
0  0 -1  5
0  1  0  0
0  0  0  1

Which is correct.

Anyways thanks for the response - I actually don't want to undo W, but apply another transformation to it (e.g. I want W' = W * R * T * inv(R) in terms of what you wrote).

Note. for my description I'm using standard math definition for multiplication - AB = A * B, AB[i, j] = sum from k=1 to m (A[i,k] * B[k, j]) - first index is row, second column. Column vs Row major terms tend to confuse people imo (even though it's just about how data are laid out in memory).