I got the restitution working now, use the exact same formula as box2d but with included separation bias like this:

// Setup a velocity bias position correction and restitution
contact.velocityBias = -this.baumGarte * dtInv * Math.min(0.0, -contact.penetration + this.allowedPenetration);
var velAlongNormal = math.vec2Dot(contact.normal, relVelocity);
if (velAlongNormal < -this.velocityThreshold) {
  contact.velocityBias += -contact.restitution * velAlongNormal;
}

But i have another issue now regarding the tangent impulse... The normal impulse method is incredible stable, even without warm starting / contact caching, but the tangent impulse causes extreme rotation - in a really weird way (I created a seperated thread for this, see: http://www.gamedev.net/topic/657825-explosive-tangent-impulses-causes-extreme-rotation/)