I recently worked on an AI program with the UDK, it was a simple AI but it make me wonder how to make a more inteligent and realistic AI? First of all, apart from the basic pathfinding, I'm kind of beginner when it comes to complex AI. But anyway, there is my answer to the first question of the OP.
1. Lets say I have bunches of enemies attacking the player, how they should take a decision to hide or attack or move somewhere else, etc...?
Each AI has a View and a Memory.
The View stock what the AI sees at the present moment.
The Memory can be the position of ammo seen before, or an order (go to this point, kill player, etc...)
Depending on the View (enemy is shooting), the present characteristics of the AI (low health), and the content of the Memory (medipack seen at a close position, commander asked to kill enemy) possibles action would be decided and noted.
- Fleeing = (very low health = 15pts) + (not many allies around = 5pts) = 20pts
- Continue attacking = (ordered by commander = 5pts) + (brave = 5pts) + (I hate my enemy = 5pts) = 15pts
The next action will be the one with the most points. Here, the Fleeing action will be the next action.
2. When they hide, how they should determine the places they will hide while making sure that the player can't see or fire on them directly?
Something similar could be used to answer the second question of the OP.
An Ai would have tactical possibilities and a position. For example :
We have 3 AI against one player. Their basic tactic is to surround the player. Each Ai are assigned with a position : center, left or right.
The left Ai will try to find the next cover in the zone right to the player. Same for the right side. The center won't move much. The goal og each Ai is to get closer to the player, or at least a good shooting distance.
To chose the cover, I guess we could add a variable determining how long the AI will stay under cover before moving. Or another corresponding to the distance without cover the AI is willing to go.
To resume the process, it would be someting like that :
1 - Decide global tactics
2 - Assign position
3 - For each AI, seek the closest cover available in the direction decided with their position.
4 - Repeat the 3rd step until getting at a good shooting distance.
Once again it's only a guess and not yet tested, . And has the poster above said, it's necessary to have
a balance between processing and ability of the AI
Because, AI can be very demanding for the process.
Sorry for the long post but I hope it can give some kind of hints or idea :)