So I've been trying to make a 2D top-down shoot-em-up game where numerous enemies chase a single player (pretty much like a zombie game), but I do not want to overwhelm the player solely through number especially due to performance issues (pathfinding, collision detection, etc.). Currently the limit is 100, and I want to find a way to give the player a challenge. After numerous amounts of testing, I find it easy to create a "rape train", where there is a whole crowd of enemies following the player behind with the player picking them off while running. I also want the AI to be "flexible" enough to give them a challenge in big maps. I do not want a "hacky" way to do this, including by making the enemies go faster or shrinking the maps to lessen running space. I thought up of having some enemies predict player movement and blocking their path or removing enemies never seen by the player for a while and spawning new ones around where the player is going. I am working on the first idea with struggle; I'll probably post on the math section for a last resort. Any other ideas?

A control system that seems good for you is assigning roaming zones (generic connected regions with holes in your map) to monsters, and let them wander randomly and chase the player only inside their zone. The zones would be potentially overlapping and usually shared among large groups of enemies. Different zone types lead to different behaviour:

• Large and open with many mooks= brutal dogpile, but you can limit which enemies participate. The player might be able to thin the enemies from outside, or to study a different route.
  
• Narrow and long passages with many mooks= your "rape trains".
  
• Small "cages" enclosing powerful threats = effectively fixed hazards, threatening the places you want in the way you want and offering the player many possibly good tactics: running like the wind, sniping from a distance, attacking from behind, taking advantage of their little random movements, inducing them to do something stupid, etc. A common pattern in Doom and Doom II.
  
• Few enemies in narrow "walkways" within a large environments = specific patrol routes.
  
• Places without enemies = empty, safe places (unless they can be shot into).
  
• Overlapping zones = convergence of enemy groups. For example, you could have zones for rooms in a building that also contain a generous portion of the corridor they open into: a player in the corridor will be attacked from multiple doors.
  

So I've been trying to make a 2D top-down shoot-em-up game where numerous enemies chase a single player (pretty much like a zombie game), but I do not want to overwhelm the player solely through number especially due to performance issues (pathfinding, collision detection, etc.). Currently the limit is 100, and I want to find a way to give the player a challenge. After numerous amounts of testing, I find it easy to create a "rape train", where there is a whole crowd of enemies following the player behind with the player picking them off while running. I also want the AI to be "flexible" enough to give them a challenge in big maps. I do not want a "hacky" way to do this, including by making the enemies go faster or shrinking the maps to lessen running space. I thought up of having some enemies predict player movement and blocking their path or removing enemies never seen by the player for a while and spawning new ones around where the player is going. I am working on the first idea with struggle; I'll probably post on the math section for a last resort. Any other ideas?

One of the ways you do this in say, pursuit and evade behaviors, is simply "predicting" future movement of the tracked agent (in your case the player). So, if you take the current velocity of the player,and just multiply it by some sort of "look ahead time" (pv * lookAhead), you can get the predicted position of the agent at X time, now this only works if the velocity doesn't change, but it does help to keep enemies from chasing the players "previous position". If you also have pathfinding built into your maps (actually set-up with waypoints) there should be some methods to find choke points and etc (which I haven't looked into, but I know that the AI system for the starcraft I bot plugin thing has chokepoint detection built in, which should be a good starting place). http://code.google.com/p/bwta/ (bwta means brood war terrain analyzer).

If you can detect chokepoints, you should be able to do something like predict the players next movement into an area with a chokepoint, and have some zombies spawn there or move ahead to there to create more tension / intensity etc.

Anyways, hope this helps.