[attachment=8361:navMeshQues.png]
As shown in the graph above, without considering the size of agent, the result path from polygon A to polygon C would be A->B->C.

But if considered the size, because of the minimum passable width ? from A to C in B is less than the agent's diameter, the detecting should avoid to across B from A to C while doing a pathfinding, and the result path should be A ->B->D->E->F->B->C.

The question is how to know which polygon or which exit border of a polygon should be avoid during a pathfinding?

You might want to take a look at the Recast and Detour library, which provides automatic generation of navmeshes. A part of the parametric structure that you populate to generate the navmesh includes a member for agent radius and another for agent height, which affect the generation of the navmesh. A solution to the problem of agents of multiple sizes is to generate multiple navmeshes, and roughly classify the agents into a fixed number of size categories equal to the number of navmeshes. This way, you don't have to complicate the actual pathfind with additional logic to cull polygons based on agent size.

Poly B in that example is horribly inappropriate anyway. It shouldn't connect to the top and bottom walls at all. Instead, the corners of the blue boxes should be connected to each other making area B represent the area between them.

That allows you to simply mark poly B as being off-limits for agents that can't use it.

Always shrink your navmesh for agent size.

We have tested this problem at work and we found that generating several navmeshes - as JT suggest - is far more efficient (less memory AND less cpu cost) than generating one navmesh that supports several agent radii.

Poly B in that example is horribly inappropriate anyway. It shouldn't connect to the top and bottom walls at all. Instead, the corners of the blue boxes should be connected to each other making area B represent the area between them.

That allows you to simply mark poly B as being off-limits for agents that can't use it.

Thank you IADaveMark. I have thought about this situation, but even poly B is generated as you say, this problem is still exsit. Or it would be an easy way to do some algorithem base on an appropriate generation?

You might want to take a look at the Recast and Detour library, which provides automatic generation of navmeshes. A part of the parametric structure that you populate to generate the navmesh includes a member for agent radius and another for agent height, which affect the generation of the navmesh. A solution to the problem of agents of multiple sizes is to generate multiple navmeshes, and roughly classify the agents into a fixed number of size categories equal to the number of navmeshes. This way, you don't have to complicate the actual pathfind with additional logic to cull polygons based on agent size.

Always shrink your navmesh for agent size.

We have tested this problem at work and we found that generating several navmeshes - as JT suggest - is far more efficient (less memory AND less cpu cost) than generating one navmesh that supports several agent radii.

Thank you for the advise of multiply navmeshes, Steadtler and JTippetts. But here comes another problem. The project I'm working on is an RTS game, so it involves dynamic addition and removing of obstacles like buildings, and it'll be complicated for me to generate multiply navmeshes dynamically in an efficient way.

I've read the pathfinding of Company of Heros, the obstacles change frequently and it uses grids not navmeshes, and that's the way I used in past too, the reason why I want to replace it by navmesh is that I couldn't find an efficient and precise way to smooth the path, it means the path should turn only at the corner of obstacles( I'm using a line of sight test to every waypoint choosed by bisection method ). The effect in Starcraft2 is what I want to achieve ( obviously the buildings in Starcraft2 are on grids, and the path of one unit is very smooth and precise ), not flocking behavior, just one unit one path. Could you give me some advises?

CoH's solution seems nice. If you dont need the information to be so dense, you can use a quadtree instead of a dense grid. That would reduce the memory, and the cost of pathfinding, line-of-sight checks, etc.

Im not sure why your string pulling is so expensive, it should be very cheap to do on a grid, as all you need to do is intersecting a vector with axis-aligned lines. If you are sure your maths are optimal, then there is also the fact that you dont need to smooth the whole path at once. You really only need to smooth up to the next "corner", its useless to smooth a whole path that is likely to change.

CoH's solution seems nice. If you dont need the information to be so dense, you can use a quadtree instead of a dense grid. That would reduce the memory, and the cost of pathfinding, line-of-sight checks, etc.

Im not sure why your string pulling is so expensive, it should be very cheap to do on a grid, as all you need to do is intersecting a vector with axis-aligned lines. If you are sure your maths are optimal, then there is also the fact that you dont need to smooth the whole path at once. You really only need to smooth up to the next "corner", its useless to smooth a whole path that is likely to change.

I think my problem has been solved. I found a precise way to find corners. And it's not expensive as I was thinking. I've decided not to use navMesh, HPA* with grids is enough. Don't need to smooth the whole path at once, got it. Thank you again.