Your number of active entities sounds painfully low... what exactly is the physics/steering stuff doing? What is your hardware target?

"..painfully low.."
Well, I'm using bullet as physics engine, have a terrain-heightmap collision handler and several hundred static collision objects (cave models) which consist of low-res model (tri-soup, 100-300 tris per model). Each entity will do collision detection only vs a small terrainmap part and only vs 1-3 of the more complex collision objects (cave models), thought the collision detection will atleast always have to handle one of the cave models, because the entity is inside its AABB. Each entity consists only of a sphere or capsule and is steered by forces in combination with one ray-cast test.

Entity vs entity collision is turned off if not one of the colliding objects is the player. For steering behaviour the closest X objects are detected and cached over several frames.

AI is based on a behaviour tree written in lua. The behaviour tree has an update frequency of 0.5 to 5 hz.

My target hardware is 2-4 year old PCs, so powerful enough.

For comparison, X3: Terran Conflict simulates several thousand entities actively across the game universe in realtime, on a commodity PC.
[/quote]
A space game has extremly good collision detection conditions. Objects are seldom in frequently contact to other objects, whereas my game each entity has to check atleast vs terrain and one cave model. I think, that the cave models are the most expensive collision tests.

Why are you doing full physics checks for entities which may be way outside any viewing/observation area?

That's the entire point of the LOD recommendation: don't do math you don't have to do, and you save a ton of CPU time. Simplify the navigation of entities using, say, navigation meshes or waypoint graphs, and you'll get a corresponding boost in performance.

Eventually you all convinced me to add some kind of level of detail including my own approach. The implementation is not finished yet, but the major hurdles are taken.
That is what I've done:
The entire map is devided into section. Each section contains information about creatures and "spawn-rate". Depending on these attributes creatures start spawning when the player is close to the according section. Entity distribution and amount is managed and limited. Addionally I now added a light-phyiscs simulation which disables collsision detection and external forces (gravity) and moves the entity along a waypoint graph. Furthermore the more expensive steering behaviours (which needs to scan the surrounding) are disabled in this mode too. This mode will be active for entities outside the players visible view distance.

I don't believe that this will enable the simulation of the whole level (several hundred entities), but it is very flexible, scaleable and the active entity limit is relaxed. The best thing is, that it enables new features like entities summoning new entities which lead often to some nasty issues in rogue-like games.

I looked at this problem years ago and you basically want a LOD (level of detail) mechanisms
that downgrades the entities AI processing when out of the players view. This can be done by
increasing the granularity size of the decision points and actions taken.

When out of view, why do pathfinding when you can simply jump to the target point (after a proper delay)
or along a simplified sequence of waypoints (often from the 'coarse' pathfinder).

It all depends if the simulation needs to be extremely accurate always or if generalized results/interactions
are good enough. If not, then 'out of sight' interactions can be generalized/abstracted to eliminate all the fine
level interactions the AI would have to calculate. The generalization might go as far as simply carrying out the
entities expected interactions with little or no positionality change on the world (except that a position is
maintained to check for reactivation when the player nears.

Some entities may simply be 'props' used for ambience which may just be a probablistic for an area
and be spawned onto the edge of the players view (radius) and then move about (but with full ability to
interact) and then disappear as the move out of the players view. When far enough from the player
(see in a distance) such 'props' might follow a routine script of actions that actually takes very little AI.
This could be daily schedules that can be calculated ahead of time and resused for routine activity
(including saved paths/routes where finesse/realisticness would not be seen thus need not have CPU
wasted on it).

Some parts of the simulation deal with groups of objects working/interacting cooperatively and these can be
abstracted to remove indivuduals interactions and replaced by simplified results applied to the world.


The hardest aspect can be when the player moves and the 'full detail' window around him causes the lower
LOD entities to have to transition back up to full detail (prefereably a little time before the player can actually
see/interact with them). The coarse granularity would likely leave them in 'safe' positions which might have
to be randomized some to make look more realistic (both position and where they are in the fine level
sequences of actions that make up their activities). When several entities 'expand' in close vicinity they
might need adjustments to be properly/naturally placed and in the middle of their expected activities.

Often more complex AIs use planners that have continuous cached states which might quickly have to rebuilt
for their current context (the fine detal past having been discarded when the entity shifted to low LOD
AI) You might have to reactivate to fine detail earlier to allow the reconstitution and the AI orienting itself
to the current situation.

Boundry Endcases would be:

Group entities (if you have them) where half are inside the player window and the rest outside -- when
would the transition to coarse LOD happen... (probably once all of them are outside or maybe split
into two groups - one full detal the other abstract).

Entities in High/fine detail/LOD may have to interact high detail with other entities also running at high level
to simulate their actions correctly (where players may see them). If entities past the boundry are forced to interact
in full detail (with those inside ) then they themselves being in high/full detail might interact with other adjacent entities
even further out -- cascading outward and likely forcing too many entities to run too much AI processing. Thus it may
take some clever logic to have the entities at the boundry interact at high level appropriately with those within the
players view and abstractly outside simultaneously.


Visually players may see LARGE entities from a long distance (depending how your graphic scheme works) and these
may have to be run at high/fine detail and be seen interacting with unseen smaller entities. DO you have a bubble of
high level around those objects....

When player has remote viewing or magnification (telescope) whatever is viewed would need to be simulated in
high/fine detail -- again possibly needing a bubble of high level AI simulation in that vicinity -- and if they can rapidly
swing their view you might have a serious problem reconstituting high level detail that fast .


Looking at such designs I even forsee a 'micro' view of a very fine level of detail immediately in the players reach
where very close interactions with small objects (or objects manipulated by entities adjacent to the player would
be done). Just another case of objects that disappear with distance and are filled in when the player is near (very near
in the micro detail case). These may not be 'entities' themselves but the nearby entities may manipulate them and interact
with the player with them when so close, and the behavior of the entities to use these objects would change as you
(the player) got far enough away (and interactions of that type cease) . The same LOD mechanism would be used to
control AI for the simulation.

If you have AI Game Programming Wisdom handy, Mark Brockington (Bioware) had an article in there about how to do LOD for a large RPG (in this case, NWN). He had 4 layers, iirc.