Oh it seems i fixed it. The relative velocity for angular rotation was incorrect - now it works :-)
Original box2d lite:
// Relative velocity at contact
dv = b2->velocity + Cross(b2->angularVelocity, c->r2) - b1->velocity - Cross(b1->angularVelocity, c->r1);
my old impl:
var tempVec = Vec2Pool.get();
var rva = Vec2Pool.get();
var rvb = Vec2Pool.get();
math.vec2CrossSV(tempVec, bodyA.angularVelocity, contact.rA);
math.vec2Add(rva, bodyA.velocity, tempVec);
math.vec2CrossSV(tempVec, bodyB.angularVelocity, contact.rB);
math.vec2Sub(rvb, bodyB.velocity, tempVec);
math.vec2Sub(relVelocity, rvb, rva);
new working impl:
var tmp = Vec2Pool.get();
math.vec2CrossSV(tmp, bodyB.angularVelocity, contact.rB);
math.vec2Add(relVelocity, bodyB.velocity, tmp);
math.vec2Sub(relVelocity, relVelocity, bodyA.velocity);
math.vec2CrossSV(tmp, bodyA.angularVelocity, contact.rA);
math.vec2Sub(relVelocity, relVelocity, tmp);
Javascript does not support operator overload - therefore i needed to make this ugly code -.-