Ok so hopefully I can explain this, I have an AABB and I have a frustum and I wanna find a minimum AABB that contains the intersection volume of the first AABB and the frustum. Does anyone know the best way to do this?

Tried a couple different ideas but they are all kinda wrong, I tried representing the AABB as a set of edges and clipping the line segments against the planes of the frustum then finding the AABB containing the resulting points. Unfortunately that approach falls apart if any of the edges are completely outside the frustum. I've worked on this for several hours now and I'm out of ideas. Hopefully someone here can help me out, thanks

ic0de said:
I have an AABB and I have a frustum and I wanna find a minimum AABB that contains the intersection volume of the first AABB and the frustum. Does anyone know the best way to do this?

The obvious solution would be a function to clip a convex polyhedra by a plane.
As we don't need to maintain mesh adjacency, that's not that complicated.
I think you'd need only a function to clip a planar polygon by a plane (i can post one, if you don't have that).
Then you would clip each AABB face by the 6 frustum plane, and make a new AABB fromt eh remaining vertices.
I don't see a need to handle any special cases with this approach.

ic0de said:
I tried representing the AABB as a set of edges and clipping the line segments against the planes of the frustum then finding the AABB containing the resulting points. Unfortunately that approach falls apart if any of the edges are completely outside the frustum.

Sounds like a good idea to avoid the redundant work and complexity of the above, which basically clips each edge twice becasue edges are shared.
But we need a solution to deal with intersection free all-in and all-out special cases, shown on the middle and right:

You can detect those cases easily by testing if the (green) frustum vertices are all inside or all outside of the (black) AABB.

So maybe you do this first, and only if those special cases don't apply, proceed with your line clipping approach.
While doing this first test, record all vertices that are all in/outside to add them to the resulting new AABB, so we handle the cases of edges which are completly inside or outside as well.

I think it should work then:

You get the blue / green circle vertices from the initial all in/out tests.
And you get the red cross vertices from the edge intersection test you already have.

Thanks for your answer! I love the diagrams, seems like it should work. However in the meantime I came up with a different approach entirely. Instead of “clipping" the AABB I realized I could move each face along its axis until it touches the plane. so basically in pseudocode:

for each face in AABB:
	for each plane of frustum:
		if all vertices are outside the plane:
			find the distance along face normal between the plane and the closest vertex
			move the face along its normal by that distance

This actually works but the whole process needs to be done twice unfortunately because some faces can move fully outside the plane in the process. But I think its actually computationally less expensive in the end than proper clipping

I will probably try JoeJ's approach too and compare performance