I am struggling with implementing collision detection for 2 shapes. The project where it is needed is constructive solid geometry. The catch is that I need a "precise" detection between 2 shapes. One is always convex and second is concave after modifying it.

I already tried to use collision detection based on bsp tree generated from a mesh (bsp tree is just a representation of a mesh in that form). In this approach I just checked whether at least one point behind a planes. If yes, then they are colliding

With second approach I tried to use polygon vs bsp tree. In this approach I checked if polygon behind or straddling. If at least one is doing so, then 2 meshes are colliding

The issue is that one of the shapes is modifiable(mesh A) and on collision I modify this mesh substruct with other mesh (mesh B). And after I update mesh with modifying mesh A with mesh B, the collison detection doesn't work anymore.

So, I am wondering, if there is other way to implement collision between mesh A and mesh B or I just introduced unnecessary complexity in my collision detection test and introduced a bug? Is there any algorithms that work on polygon soup A and polygon soup B? Or maybe there could be algorithm to calculate hull of meshes so that I could use gjk algorithm or sat? Note that mesh A and mesh B only consists of polygons and I don't think there is possibility of using bounding boxes for this. Sorry for some really bad English here. Thanks

You should probably decompose the concave shapes into convex pieces and process them independently, because you want to use algorithms that handle only convex shapes like BSP trees (they represent regions as an intersection of halfspaces).