Sometimes when I want to open less node in the opened list,

I would give the higher g cost node a higher rank, but the result is less nodes get opened,

the problem is that the agent is unable to find alternative path.

This problem is indeed a problem.

Thanks

Jack

Post code.

Many different things can be wrong; your problem description sounds like typical A* errors, like wrong node priorities or inadmissible heuristics or both, but they would normally cause the search to return a suboptimal path; additional or worse defects are needed to fail to find a path.

Not finding a path, means you have exhausted the open list completely. (Would be worth checking that, probably.)
The reason for that is because the A* algorithm will continue expanding open nodes until a) found the destination, or b) running out of nodes to expand.

So if you are indeed running out of open nodes while you see a solution, you must be missing some nodes that should have been in there. If you can point out one, set a break point on the neighbour that should add it, and find out why the missing node is not added.