Lets say I concave triangular mesh surface, M, and convex submeshes, sMi, each of which are generated from the surface of M (so that any shared triangles can be considered identical). Submeshes can overlap eachother.

Lets say we need to find the total area of M that is not part of over of the submeshes.

I would do this by simply iterating over every submesh, and flagging each corresponding triangle on M. Then simply summing up all non-flagged triangles.

This is easy. My problem is a little more complicated.

My submeshes contain not only fully corresponding triangles, but also triangles that have been subdivided into partial triangles. They are generated from circular projections from the centre point normal.

Here is an example of a submesh, with black lines being the triangles from M and the green lines for the cuts of the subdivided triangles. The black circular outlines could be ignored.

Remember that these submeshes can overlap, so therefore the sub triangles could also overlap.

For example, this particular triangle is overlapped by two submeshes, blue and green (purple is the union of both).

Does anyone know of an approach I could look into for this type of problem?

I would probably decompose the original triangles in smaller triangles when the submeshes are added and then use the algorithm you have already outlined.

I would also do what apatriarca suggests, but here's an alternative: Use the inclusion-exclusion principle. That means:

Area of the union = (Sum of the areas of the pieces) - (Sum of areas of two-by-two intersections of the pieces) + (Sum of areas of three-by-three intersections of the pieces) - ...

I don't think that's particularly useful in your situation but, since we are talking about computing the area of a union, it's probably a good thing to be aware of.