For some reason your boxes aren't pushing apart from one another even though there seems to be significant penetration. Are you handling the bias term appropriately? With no friction you should not have any torque if you land one box onto another from such a short height, if you have enough solver iterations.

Try checking your code against the code in Box2D Lite. There you have a full working example and should be able to identify any hard to catch mathematical errors (like forgetting to normalize something, or scale something).

I'd look at your source code but just don't have the time, so if you're stuck you have to keep trying new avenues of debugging.

Edit: Oh you disabled stabilization? Well then of course you're going to get results like this. If one point is solved without proper convergence you will get a divergent simulation, and without bias it cannot get enough energy to correct the slightly uneven contact points.

Usually when debugging this kind of code it's good to disable features and isolate problems, but that won't help you if you don't understand how this will impact the simulation (as in since you don't know what to expect you don't know what is correct).

Even with re-enabled stabilization the side moving is still doing its thing - but it takes a lot longer to move the boxes.

Also i have 10 solver iterations whichs should be enough to get it stable for such a simple case.

I uploaded a video showing the contact generator results:

It looks totally fine - all points even when the reference face is on B, points are projected back to A.

Looks good from what I can see there.

After looking at the r1 and r2 points in box2d lite - this problem is also present when warmstarting is disabled, but its visually not noticeable because the entire coordinates are tiny compared to my pixel based system (30 times smaller than my implementation) and if i enable warmstarting in box2d lite the issue is totally gone (r1 and r2 are always the same).

So basically to solve this issue i have to implement warm starting as well - which should be fairly easy, because i already have saved the previous contacts.

Also i should scale down my world entirely by a factor of 30 or 50 - so that floating point errors gets tiny so that its not noticeable at all ;-)

A lot of work for the renderer i must do here... arghhh.

Incredible!!!

Applying warm starting does solving this issue completely - even with my pixel based coordinates.

Thanks for all the tips i got from you guys ;-)

It was too early to get happy... i tested stacking boxes, the problem is still there :-(

This time.... i finally solved it completely.

It was a bad combination of minimum penetration distance (slop) and baumgarte parameter. Now it stabilizes perfectly.

// NOTE: Baumgarte and minimum penetration must fit together
public static final float MIN_PENETRATION_DISTANCE = 0.1f;
public static final float BAUMGARTE = 0.3f;