Regarding GameBryo there is a reason that Trion and Bethesda use a heavily modified version of GameBryo. I know that Trion only uses some of the tools (which were to get them up and going while building their own engine) and Zenimax Online is not using it for their MMO. The architecture of this engine is old and stale and the main reason Bethesda stayed on it for single player games is because they built on it over many years and games. GameBryo is very similar to the Wild Magic engine in architecture although it has evolved slightly over the years.
Actually, from what I've heard Bethesda did not significantly modify Gamebryo until they created "their own" engine for Skyrim, which highly resembles Gamebryo. Trion has stated multiple times that RIFT was built using Gamebryo engine. I wouldn't imagine them lying about not having their own engine for the game. I don't know where you get your information from, but I'm suspicious considering the only way you could actually KNOW they were using a heavily modified version of Gamebryo is if you worked with them or had seen the source code yourself. Your definition of "highly modified" also probably is different from mine. Of course they're going to modify it or add wrappers over some classes to better suit their needs. The ability to do so fairly directly is a big advantage Gamebryo has over other game engines such as Unity.
As for the architecture being "old and stale", this is an opinion. If I were investing millions into a game, I'd definitely prefer a traditional and proven approach rather than a radical and risky one especially for a game engine. If the company wants to take more risks, they should develop their own engine in favor of unproven solutions.
Regarding UE2.5/3 it has been used by numerous companies in the past but you *need* a source license and a lot of work in order to get anything up and running for a massive online game. Also I really don't see any benefit to using UE3 over something tailored directly to the genre.
I'll agree that it would definitely take a lot of work to get it running with an MMO, and you're right that UE3 isn't the best solution if you want to have support for population condensed into a small area. However, I've seen some MMORPGs that are more instanced and could be developed without support for more than 64 players in one area. This isn't an ideal solution, but certainly a possibility.
Star Wars TOR (heavily instanced / story based), The Repopulation (complete open world sandbox), Faxion Online (traditional type MMO) have all been built on HeroEngine and as I mentioned in brackets they are extremely different types of games. I'm not sure what you mean by "too managed"? The tools are extremely in-depth and give you everything to build, run, and manage an online game. Also note that Zenimax Online licensed HeroEngine for The Elder Scrolls Online just for the use of the toolset so they could get started on world building, design iterations, etc while they build their own custom engine.
I apologize for being unclear. By "too managed", I mean the scripting language seems like very high level code in contrast to a much lower level language like C++. The reason I said "seems", was because I haven't played around with it much, but I have played SWTOR for awhile and in my opinion the controls and overall gameplay is quite clunky compared to games like WoW or RIFT. I indeed have not played The Repopulation or Faxion Online. I'm simply stating my opinion based on having played SWTOR and having worked a bit with the engine in the past.
Another great choice is to use a combination of Unity and a socket server like Photon (definitely the best to pair with Unity). You can download the free version of Unity and Photon and get started right away... plus they even give you a small sample to get you started quickly before you have to dig into the docs. Unity has been used for MMO games such as Battlestar Gallactica, FusionFall, etc... so I don't see why there is an issue there either.
Unity is a good engine for browser clients. That's about as far as it goes from what I've seen. Both examples you cited only use Unity for their browser client. In my opinion, if you're going to have 64+ players visible in one area, you're heavily limiting your performance by forcing players to use a web browser. That being said, it may be a good idea to use Unity for a web client, then use another or your own for other platforms. The web client shouldn't be the main platform for a MMORPG in my opinion though.
Actually HeroEngine and Big World have a massive number of AAA MMO games built using their technology and the above you listed really only have a couple (being a few that use a modified UE3... the only MMO I count for using GameBryo is Dark Age of Camelot as the others use a version that isn't even recognizable. Meanwhile UE3 hasn't powered that many either and the ones that did used it had to heavily modify and add to it as well.
Your definition of AAA is very different from mine. The only MMORPGs I've seen Big World's engine produce are Asian(mostly Chinese it appears). I've tried a couple of these said games, and they seem clunky with poor production values typical of Chinese MMORPGs. I'd like to see some proof that RIFT doesn't use Gamebryo heavily. There have also been successful Korean MMORPGs (definitely) using Gamebryo, but I wouldn't count these as AAA. You're right about UE3, but I would still prefer it over Big World or Unity for developing non-browser clients due to the quality of the engine coupled with the successes of RPG games using it.
You seem bias against enterprise-level solutions in this discussion. No development company with a large enough budget to make a decent MMORPG is going to use Unity for an MMO as anything other than a browser port. Unity is great for many genres of games, but MMORPGs doesn't seem to be one of them. I'm not implying that it's impossible to make a multi-platform MMORPG in Unity, but it's definitely unreasonable.