I've tried to implement some basic tangential friction (Coulomb) now and it's looking decent with small stacks and forces.

However the rotational velocities quite often become exaggerated and objects also get stuck balancing on corners (oscillating back and forth).

Maybe tangential friction doesn't account properly for corners? This is also without warm starting (it doesn't seem relevant here?).

Things I have checked:

• collisionNormals are correct. (0,1,0) when colliding with floor
• When object balances on a corner there are 1 contact point

Video examples:

Balancing on corners: https://i.imgur.com/FqGa2nl.mp4

Exaggerated rotational velocity: https://i.imgur.com/IGWGujz.mp4

Friction implementation (however this rotation problem exists without friction and without PGS iteration):

// --- Tangential (friction) impulse calculation ---
// Isolate the tangential component of the relative velocity
glm::vec3 v_normal = glm::dot(relativeVelocity, contact.normal) * contact.normal;
glm::vec3 v_tangent = relativeVelocity - v_normal;

// Check if there's any significant tangential velocity
if (glm::length(v_tangent) > 1e-6f) {
    // Normalize the tangential velocity to get the direction of the friction
    glm::vec3 tangent = glm::normalize(v_tangent);
    
    // Project the relative velocity onto the tangent
    float v_t = glm::dot(relativeVelocity, tangent);

    // Compute the effective mass in the tangential direction (similar to the normal direction calculation)
    float effectiveMassTangent = 1.0f / (objA.invMass + objB.invMass +
        glm::dot(glm::cross(cp.rA, tangent), objA.inverseInertia * glm::cross(cp.rA, tangent)) +
        glm::dot(glm::cross(cp.rB, tangent), objB.inverseInertia * glm::cross(cp.rB, tangent)));

    // Calculate the preliminary tangential impulse
    float J_tangent = -v_t * effectiveMassTangent;

    // Apply Coulomb's friction law: limit the tangential impulse
    // Use the change in the normal impulse (deltaImpulse) as a reference.
    float maxFriction = 0.3f * fabs(deltaImpulse);
    J_tangent = glm::clamp(J_tangent, -maxFriction, maxFriction);

    // Build the tangential impulse vector
    glm::vec3 deltaFrictionImpulse = J_tangent * tangent;

    // Apply the friction impulse
    objA.linearVelocity -= deltaFrictionImpulse * objA.invMass;
    objA.angularVelocity -= objA.inverseInertia * glm::cross(cp.rA, deltaFrictionImpulse);

    objB.linearVelocity += deltaFrictionImpulse * objB.invMass;
    objB.angularVelocity += objB.inverseInertia * glm::cross(cp.rB, deltaFrictionImpulse);

    converged = false;
}

So my problem with strange rotations was actually due to my implementation of quaternion rotation being wrong.

Now to implement warm starting!