Hello all. I am working on pathfinding and I am using navigation meshes. I am using A* and have it working. However, as I'm sure most of you know, there's a 'jagged-ness' to the calculated path. I am using the vertices of the various navigation meshes to calculate my path (hence, for me, the jagged-ness). Now the beauty of navigation meshes so that any point on a nav mesh to reachable to any other point on the same nav mesh. So I am trying to put two and two together by using raycasting/line-of-sight/whatever to smooth out my path.

I don't know how.

My current plan is to post process my calculated path to make it smoother. This will be done by taking the starting point and checking if it has a line-of-sight to the goal, if not check if the starting point has a LoS to the point before the goal and so on. When a LoS is found a path from the start to the point is created, and the process starts again with the found point being the new 'starting point'. Sounds good, but I have a problem with the line-of-sight part.

I don't know how.

My current plan (regarding the LoS issue) is to calculate a polygon 'outline' that covers the navigation meshes (IE: I'm taking several (overlapping/connected) polygons and 'adding' them together so any overlapping parts are removed and I have a single silhouette polygon). Once I have that, I can simply check if a ray from point-to-point crosses said polygon's edge, if it does then there's no LoS. Of course, I have a problem calculating the outline polygon.

I don't know how.

So my question is: How do I calculate the polygon outline from set of overlapping/connected polygons?

Or, perhaps even, is this dumb and is there a better way to do this?

What you want is http://en.wikipedia.org/wiki/Boolean_operations_on_polygons with the union operations. It's definitely not an easy thing to implement though, so if you can find a way to do without this, it's probably preferable.

I'm by no means an expert in this area, but I think you are thinking about this a little wrong. The advantage of using a navigation mesh is that the polygons you are dealing with are convex, so it is actually very simple to work out if there is a LoS from polygon A to polygon B. If you merge your polygons the way you describe, you will have a non-convex polygon which becomes far, far harder to work with. In fact, if I were trying to work out a LoS across a concave polygon, I'd probably approach this by first decomposing it into convex polys which would be the reverse of what you are proposing :)

As I understand the nav mesh, you step from one polygon to the next using a nominal position - say, centre of connecting edges - to confirm the path exists and to form an initial rough path. You then apply smoothing to this generated rough path to minimize its distance. I'll leave the advice on how that works to people who have actually implemented it.

I've changed my plan of attack to more inline with Theta* pathfinding (not sure if that's an actual thing or just the name the guy came up with, http://aigamedev.com/open/tutorials/theta-star-any-angle-paths/). However, I still need to determine if two points have a 'line of sight' with each other (the entire vector is in a nav mesh, though not nessessary in the same nav mesh).

Any ideas?