My reaction fire work like this:
1. You save timeunits (TU) so you can use reaction fire for a soldier
2. He will fire if enemy comes within line of fire at reduced accuracy and increased TU-cost. Normal damage.


But how to avoid a player from setting up all his soldiers in a good (and concentrated) position, activate RF for everyone and simply wait until enemy comes into the trap, one by one. It seems very hard to code AI that would not be weak to such a "tactic".

Any tactical approach you don't want to prevail (in this case dedicating many soldiers to ambushes in order to take advantage of carefree AI soldiers who wander into the crosshairs) should be dangerous and not too useful.

To make ambushes dangerous, you can include slow-moving long range snipers who can pick off enemies whenever they stop too long in one place and maybe area attacks that can decimate clustered "firing squads" very effectively.

To reduce ambush usefulness, less dumb AI (i.e. AI that doesn't get shot without purpose) is unfortunately the top priority. What exactly do you find difficult?

With perfect information, which might or might not be a form of cheating depending on your game's rules, enemies can use pathfinding to simply avoid walking into positions where they could be shot, treating locations covered by enemy weapons like obstacles. AI soldiers should avoid dangerous zones and pop out of cover when absolutely necessary or when they can attack advantageously (i.e. when they can hurt the enemy more than the enemy can hurt them, for example 2 vs 1 with the same weapons).

Since you are in control of maps, you can also avoid really good choke points: concentrating forces in one place should allow the enemy to take another route.

I'd go with tunnel vision. The old Full Spectrum Warrior game had a system where you could tell a guy to watch a certain direction and he'd be very effective in a narrowly focused cone. Just as powerful as active engagement, but not as versatile.

Restrict RF to a direction that the player chooses at the end of the turn. Good for watching a corner or vector for incoming threats, but not a win solution for the enemy's entire turn. Make the player do their work on their turn, but let them hedge their bets when the enemy has the initiative.