AhmedSaleh said:
if (triangle_triangle_intersection_check(triB, triA))

Basically you need a function similar to this check, but clipping one triangle into multiple sub triangles, like the pictures show.

But that's too complicated for me to write code out of my head. It's similar to _cg_plane_split_polygon(), but we do not clip with a plane but a triangle, so the intersection becomes a line segment, not just a infinite line. It's thus more complicated. The most complicated case should be this:

You see the intersection (bold red line) is a segment inside the black triangle. The black triangle becomes 5 new sub-triangles here. 
(Some Detail: You also want good tessellation, meaning you want to avoid small angles in triangle corners. This helps to avoid bad triangles and precision issues.)

We then mark the original larger (black) triangle to be inactive (or delete from the list if you prefer), and add the new smaller (blue) triangles to the list.

As we continue to iterate other triangles, those previously added sub-triangles may be split again, and being replaced with even more and smaller sub-sub-triangles.

I can't describe it any better. If there is confusion left, you need to describe in detail which things are unclear.

@joej

Now I get the intersection segments between the two triangles, and I get the following result using the following code:

	int num_indices = 0;
	for (size_t i = 0; i < tri_mesh_b.vertices.count; i += 3)
	{
		musa::Triangle triB{ tri_mesh_b.vertices[i], tri_mesh_b.vertices[i + 1], tri_mesh_b.vertices[i + 2] };
		stack.push(octree_a.root);

		while (!stack.empty())
		{
			cg::Octant* node = stack.top();
			stack.pop();

			if (box_box_intersect(musa::triangle_bounding_box(triB), node->region))
			{
				auto vertices = node->faces;

				for (size_t a = 0; a < vertices.count; a += 3)
				{
					musa::Triangle triA{ vertices[a], vertices[a + 1], vertices[a + 2] };

					if (triangle_triangle_intersection_check(triA, triB))
					{
						mn::buf_push(self->modelIntersection.points, triA.p0);

						mn::buf_push(self->modelIntersection.indices, num_indices++);
						mn::buf_push(self->modelIntersection.points, triA.p1);

						mn::buf_push(self->modelIntersection.indices, num_indices++);
						mn::buf_push(self->modelIntersection.points, triA.p2);

						mn::buf_push(self->modelIntersection.indices, num_indices++);

						musa::Intersection_Segments segments_ = musa::triangle_triangle_intersection(triA, triB);
						
						for (auto& segment : segments_.segments)
						{
							mn::buf_push(self->modelIntersection.points, segment.start);

							mn::buf_push(self->modelIntersection.indices, num_indices++);
							mn::buf_push(self->modelIntersection.points, segment.end);

							mn::buf_push(self->modelIntersection.indices, num_indices++);
						}

						
					}

				}

				for (auto& n : node->octants)
				{
					if (n == nullptr)
						continue;
					stack.push(n);
				}
			}
		}
	}
	

I think it's worth to work on some better visualization, e.g. drawing wireframe, but offsetting edges a bit so it becomes more clear what is going on.

To me it looks like this:

It shows me adjacency of a half edge mesh, so looks pretty complicated, but this often helps me to understand bugs much faster.

@joej

Thanks for your drawing.

I would like to know the inside algorithm here

musa::Triangle triA{ vertices[a], vertices[a + 1], vertices[a + 2] };

					if (triangle_triangle_intersection_check(triA, triB))
					{
						mn::buf_push(self->modelIntersection.points, triA.p0);

						mn::buf_push(self->modelIntersection.indices, num_indices++);
						mn::buf_push(self->modelIntersection.points, triA.p1);

						mn::buf_push(self->modelIntersection.indices, num_indices++);
						mn::buf_push(self->modelIntersection.points, triA.p2);

						mn::buf_push(self->modelIntersection.indices, num_indices++);

						musa::Intersection_Segments segments_ = musa::triangle_triangle_intersection(triA, triB);
						
						for (auto& segment : segments_.segments)
						{
							mn::buf_push(self->modelIntersection.points, segment.start);

							mn::buf_push(self->modelIntersection.indices, num_indices++);
							mn::buf_push(self->modelIntersection.points, segment.end);

							mn::buf_push(self->modelIntersection.indices, num_indices++);
						}

						
					}

What I don't understand

If I want now to do triangle to triangle intersection, I get segments as result, I have to triangulate them ? I didn't understand… the polygon clipping is still not working.. as you see there are interior triangles that are not included in the intersection mesh in the above reply

AhmedSaleh said:
I get segments as result, I have to triangulate them ?

Yes.

I don't know what your triangle_triangle_intersection() is doing. I'm confused because it seems to return multiple segments, although the intersection of to triangles can give only one segment.

However, i guess you get the red bold intersection line from my latest handdrawn picture, and now you have to compute the blue edges.
That's maybe the point where you want to setup only two triangles (not two whole meshes) to limit confusion. I often use ImGui to change things like vertex positions at runtime in such cases, to test things out. It also helps to draw all possible cases on paper at first.

@joej

I get line segments from triangle to triangle intersection, the line segments are all x = 0, and they have y, z with values. I get just one segment per intersection. so I have to save them somewhere and triangulate them ?

So I triangulate them using ear clipping for example, what should I do after that ?

AhmedSaleh said:
So I triangulate them using ear clipping for example

Yes. (Though, i think the problem is too simple to require earclip)

After that, the initial triangle can be disabled or deleted, and the new tessellated triangles should be added to the mesh. This way we can continue to clip stuff further, and at the and we see the intersection edges in both meshes.

Maybe the algorithm i proposed could be simpler if using just one global stack per mesh, to deal with removing old and adding new triangles. But then you still need to maintain relationship to octree as well, adding and removing them here too.

@joej The current triangle_triange intersection which results in one line segment, it returns a line segment which has 3D line start and end, and the ear clipping triangulation accepts 2D points…

The current code ignores the blue one

don't know why

@joej

	for (size_t i = 0; i < tri_mesh_b.vertices.count; i += 3)
	{
		musa::Triangle triB{ tri_mesh_b.vertices[i], tri_mesh_b.vertices[i + 1], tri_mesh_b.vertices[i + 2] };
		stack.push(octree_a.root);

		while (!stack.empty())
		{
			cg::Octant* node = stack.top();
			stack.pop();

			if (box_box_intersect(musa::triangle_bounding_box(triB), node->region))
			{
				auto vertices = node->faces;

				for (size_t a = 0; a < vertices.count; a += 3)
				{
					musa::Triangle triA{ vertices[a], vertices[a + 1], vertices[a + 2] };

					if (triangle_triangle_intersection_check(triB, triA))
					{
						mn::buf_push(self->modelIntersection.points, triA.p0);

						mn::buf_push(self->modelIntersection.points, triA.p1);

						mn::buf_push(self->modelIntersection.points, triA.p2);


						musa::Intersection_Segments segments_ = musa::triangle_triangle_intersection(triB, triA);



						mn::buf_push(self->modelIntersection.points, segments_.segments[0].start);

						mn::buf_push(self->modelIntersection.points, segments_.segments[0].end);


						musa::Intersection_Points pts = musa::segment_triangle_intersection(segments_.segments[0], triA);
						mn::buf_push(self->modelIntersection.points, pts.points[0]);

						
					}

				}

				for (auto& n : node->octants)
				{
					if (n == nullptr)
						continue;
					stack.push(n);
				}
			}
		}
	}

Seems all fine : ) 

(But i still see no tessellation happening.)