I have been playing with a prototype and I'm trying something like this:
There are expansion limits, based on the distance from the homeworld and the number of planets controlled. BUT, the trick is, these restrictions apply to AI races only, not to the player :) The player, is "destined" to build a huge empire, there are some inconveniences with size as well, but these are, unlike for AI, soft limits.
Also, some weak AI races don't have such restrictions, so they can spread everywhere, but due to their pathetic nature (space nomads and the like) they do not pose a real threat.
I addition, the AI value planets near their homeworlds much more and will be simply unwilling to conquest too far, unless left no (better) choice.
Example:
* The Hive is a race of insectoids, there are 3-5 overminds that reside on various planets, they can spread the psyhic control over minions at 3 systems distance. Therefore, they are unable to expand too far from their planets with overminds (and because there are more than one overmind it does not look like a boring circle). In addition, the Hive will fight much more desperately for planets near overminds.
* Methane breathers do not exist at the beginning of the game, they come in the middle game via the galactic outskirts. Then they start colonizing planets nearby their "warp point planets". They are not limited by distance, but by the number of planets (they are doing it slowly and don't take more they can chew), they simply compare the current turn number to number of planets they posess and it determines their aggressiveness. Even though they are powerful they try to avoid confrontation even with weaker ones (they prefer uncolonized planets).
* Space rodent's monarchy, they are the most traditional ones (they can be negotiated with and recognize the player's Empire). They have a homeworld and up to 6 "core planets" that are exactly 2 jumps from each other, then they try to colonize planets near their core planets (at 2-3 distance) prefering to maintain uninterrupted route connecting all core planets. They have no hard distance limits and can expand quite far from core planets (if they can't spread equally) but they do not feel strongly about protecting these (and you can even negotiate buyout of the far away ones, also in some cases these far one can willingly seceed and join your Empire if within your cultural influence). They do not have a hard limit on number of planets but the more they have the more likely is a civil war and other internal calamity that will trimm their kingdom).
* The Elders, declining race, they should extinct (negative breeding rate) around mid game. Very powerfull, passive, hedonistic, do not expand, do not conquer. They have several clusters of planets in like 3 parts of the galaxy (and many planets owned by others are planets previous owned by Elders), these planets die out over time leaving sometimes interesting artifacts and infrastructure (the player can speed up the process by proper use of military :)).
* Parasites, the most odd ones, they have exactly one planet. Once every 100 turns they go on a mating spree infesting a huge part of the galaxy ("conquering" planets one by one) and then after a while they abandon these planets and return to their homeworld (leaving decimated population and ruined cities - the infested planets then are able to rejoin their original owner, if someone is still alive there). After 100 turns the cycle repeats (until someone destroys their homeworld which is very hard).
Not sure how it will work in practice, but on a quick simulation of the galaxy map it looks OK... Note the player starts AFTER these aliens are run for several turns with fast colonization mode (so all/most of those aliens have many/maximum planets already).
Anyway, comments, more ideas (I'm also gathering ideas for alien races and their mechanics)?