Net_ said:
Edit: Just realized that you can just use the existing rectangle and the AABB of the hole to figure out the new rectangles, no need for re-iterating over the tiles. I'll think more about this.
Usually it requires a spatial acceleration structure, like an Octree or BVH, implementing a range query.
A player AABB would be such range. We find all rectangles, objects, or whatever stuff that intersects the given range quickly, and then we can perform collision detection or user editing on the returned set of objects.
In physics simulation terms, that's the ‘broad phase’ collision detection system, meant to find potential colliders quickly with large sets of objects. It's a bit like frustum culling for graphics.
The actual intersections in detail, and what you currently worry about, is the ‘narrow phase’ part of the collision system.
The simplest way for a broad phase is a regular grid, making the query trivial to implement and O(1). If you can't do this, usually because a 3D grid takes too much memory, you also need to think about a spatial acceleration structure.
Net_ said:
I would still have to build the mesh myself out of my collision primitives, right? Then I would be able to feed that mesh into something like bullet? The mesh building part is what I'm concerned about.
Usually you do not need to merge all your primitives into a single mesh.
Dynamic (movable) primitives supported by physics engines are:
Boxes, spheres, capsules, ellipsoids, cones, cylinders. Convex polyhedra. Height fields.
Static primitives are:
All of the above, plus a concave triangle soup.
You can compose a scene from overlapping primitives, meaning they can intersect.
There is no need to build a surface mesh, removing all the internal intersections from overlapping primitives.
So, you could build the 3 primitives from your image example as convex meshes, and create a scene from many instances of those building blocks. It should work. There might be unexpected problems from discontinuities though. E.g. if you place two boxes adjacent to each other, collisions might happen on the gap between them. So i would use capsules for players, not boxes, to smooth out the discontinuities.
I'm also optimistic regarding user edits, so primitives could be added and removed at runtime.
However, finding the ideal physics engine for your needs and learning how to use it takes some time as well ofc., but it's nothing compared to making your own physics and collision systems if you need convex polyhedra.