Hi people, I've coded my physics engine (purely for educational purpose). I use SAT for collision test, and use simple shapes (spheres and obb so far) only. Recently I added support for sphere - triangle meshes test. It worked beautifully. Then I coded the obb - triangle meshes test. It works too, but sometimes the box got "stuck" to the edge of the triangle. I've been pulling my hairs for now but recently I noticed that it usually happens between the box and the edge of the triangle which is "concave" (if we take into account the neighboring triangle).

Here's the picture that better depicts the problem (the dot is a 3D edge viewed from the side) :

For the triangle meshes, I only make a simple data structure like:

TriMesh:

-list of vertex

-list of triangle

Vertex:

-xyz

Triangle:

-vertex_id[3]

So am I right that I could simply ignore that edge? but what is a criterion in which an edge should be removed? what should I add to my data structure? I thought that only the edge that is "concave" that should be removed, but I'm not sure

Well, you can mark up your triangle mesh. Check the adjacent faces of a shared edge and mark edges as concave, convex or flat. You can also mark concave vertex. A vertex is concave if at least one adjacent edge is concave. When you check the edges, you simply skip over those that are concave.

I don't think that concave edges should be skipped though. In your example image just move the the lover right vertex of the box such that it is exactly coincident with the concave edge and pressing into it. If you ignore the concave edge you can be pushed out of the world. I assume that have wrong contact geometry and I would debug this first.

You might be getting normals from the edges that point very harshly towards other triangles. One solution may be to come up with a scheme that clamps potential collision normals within an acceptable range. Otherwise we might need more details about the specific problem you have, like debug render of contact points + normals, along with a GIF animation of the problem.