I was thinking about how the AI respawns in mmos, and also how the equipment never matches what they're wearing. One problem is scale. The populations of mobs are too small to reproduce in anything resembling a 'realistic' way,

In medieval societies (which most fantasy is based on), there really wasn't much of the population that engaged in military type activity. Unlike today, virtually everybody had to grow (to some degree) their own food.

As a general number (it varied form place to place and from time period as well), it would take about 9 people to grow enough for 10 people. This means that around 10% of the population was not involved in food production. However, out of this 10%, not all of them would be in military service.

So think of this in terms of Mobs. The creatures you see represent significantly less than 10% of the population (probably around 0.5% to 1% of the population for a war like population). So for every creature you fight, there would be between 100 to 200 more creatures somewhere (of course this does not apply for undead ).

Also most soldiers where never professional soldiers, but average peasants that were drafted or serfs that were required to serve in the army, or just the average citizen trying to defend their homes.

In terms of dungeon mobs, where the mobs have made their home there, you might see close to 20% of the population able to fight (50% female, 30% elderly or children which leaves 20% capable of fighting).

As you also would usually want your best fighters in in defensive locations (to better defend them and also to protect your assets) then the tougher creatures will be near the home town/fort/nest/etc of the mob. But, if they were making attacks, the stronger fighters will more likely lead the attack as it would give the attack the best chance to succeed.

Now, knowing all that, we can work out a system for simulating monster populations.

If we assume that any mobs not part of the Monster Nest represent around 1% (at best) of the total monster population, then this gives us an estimation of the total population of the monsters (which we can use to determine the total strength of that particular Monster Nest for other parts of the simulation).

Mobs encountered in the Nest itself will be a combination of the strongest (Boss types) and a quickly assembled draft of combatants equal to about 19% of the population.

The other 80% are non combatants and can be thought (or even shown) to just run for cover and try to get away form combat. If the nest is defeated, these can be thought to scatter and regroup where they can as a new population (and then work out the 20% of combatants from that - more on that later)

So at most, a complete destruction of a monster nest should not kill more than around 20% of the population.

If we break the world up into regions (essentially a site like a cave/ruin/etc and the area around it), and each region can sustain a certain number of monsters then we can allow each region to host a particular monster type. Over time this population will increase (breeding form the 80% of non-combatants). If the region can't sustain the population, then some of them will try to attack an adjacent region and defeat the monsters that live there. If they succeed, then the monsters push the existing monsters our and start a new colony (say 10% of the population). If they fail, then the survivor (if any) return back to the nest and the diminished population might be sustainable (if not more raids will occur).

If a population is pushed out of their location, then they will be thought of as a homeless population and will not be allowed to breed, however they can still be allowed to make raids and attempt to conquer a new home (they will still follow the 80/20 rule of 20% of the current population as combatants but they can have the 20% as attackers rather than the 1% of an established population).

Thus we can have a natural flow of monster populations and habitats in the world. It takes a fair amount of effort for a nest to be wiped out by an invading monster population (excluding the effects of adventurers ), and as this occurs, the ownership of the land will switch back and forth between the various attackers and defenders of a region.

Now, you can do some things to make it more interesting:

One is to have several ecological niches in each region (including some that are created by certain monster types, eg predator prey types) and allow interactions between them. Then each monster species can be designed to fill that niche (or niches).

For example: Just say you have a Browser type monster (one that can gather their own food from the environment - eg eats animals or raids nearby farms), then you could have a Predator type that attacks other animals and monsters for their food. Then you could have a scavenger type that lives off the waste of the others (eg: eats the corpses of other animals - or an exotic one that eats treasure).

So the more monsters that are around, the more predator type monsters can exist (if they wipe out the other monsters, they won't have any food and as the region can't sustain them their population will fall - either through starvation, migration or death in raids).

Another idea is to have special monsters that can only live in one area. These would be the really big and dangerous monsters like dragons, trolls and such. These would only have 1 or two monsters allowed per region. They would also have the ability to depopulate an area over time (forcing them to keep attacking and moving on to nearby areas to survive). They would act as normal monsters (having a nest) but as they are single monsters they wouldn't have the population effect of the 80/20 rule. Also when they successfully raided and defeated the monsters in another area, they would move into that without leaving any population behind (or only occasionally doing so). This way you can have your big monsters remain rare and still have them exist as a dynamic element in the world.

Using a system like this can lead to an interesting world with dynamic monster locations and populations. Players will have to search for the Dragon's lair and there will be effects on the world's ecosystem as both players and monsters raid each other.