On the note of "web-app stuff applied to an MMO server", has anyone checked out NetEase/Pomelo (an MMO engine built on top of Node.js)? For what I've seen it has a pretty straight forward way of encapsulating distributed servers (although it seems to still suffer if too many players congregate in one area). Sadly, I haven't been able to find any real analysis apart from these slides: http://www.slideshare.net/xieccy/pomelo-jsconfasia So I don't really know how it handles server interactio and how would a physics simulation would run in there.

offtopic:

I bet through some clever trickery and using player locality you could engineer a system of virtual boxes or spheres of players that overlapped and allowed you to grow or shrink on demand.

Awesome, I recall thinking about something like that when I "dream of making an MMO some day" in my teen years. I'll have to try it out when the day finally comes.

I'll reiterate my earlier cynicism: in the realtime massive distributed space, until you have millions of players running on Solution X, you can't believe any claims about whether or not Solution X is actually feasible.

Even the leading experts in this stuff have a hard time predicting how well something will scale. If it hasn't been shown to successfully host a nontrivial simulation with a very large number of concurrent players, it may as well not exist.

I'll reiterate my earlier cynicism: in the realtime massive distributed space, until you have millions of players running on Solution X, you can't believe any claims about whether or not Solution X is actually feasible.

Even the leading experts in this stuff have a hard time predicting how well something will scale. If it hasn't been shown to successfully host a nontrivial simulation with a very large number of concurrent players, it may as well not exist.

Aye, and the problem is that until you actually do hit that mark (in business or games) you can't be sure your solution will scale. This is one of the reasons businesses and game studios stick with existing solutions, as they have known scaling, and constraints. Trying to sell a business on a new model without having actual real world products backing it isn't easy.

That being said, there's no reason not to try out new ideas... provided you have have a great deal of theory and similar to back you up on whether a particular idea is good or not, and possibly even some trial simulations as well (AWS is great for these, cheap enough to to make slamming a system with 10-100k clients, than waiting till you hit that number of actual clients and find out your system can't handle it).

I'll reiterate my earlier cynicism: in the realtime massive distributed space, until you have millions of players running on Solution X, you can't believe any claims about whether or not Solution X is actually feasible.

Even the leading experts in this stuff have a hard time predicting how well something will scale. If it hasn't been shown to successfully host a nontrivial simulation with a very large number of concurrent players, it may as well not exist.

I could say this is very true (it is not even cynism but calm understanding) and it spans over whole programming, prepare for some words of wisdom:

though programming seem to be very logical activity, very big amounts of factors are always outside of your logical enfolding, in other words there are many factors which impacts on the outcome of your work but you do not know how they work * (often you even dont know what they are) - you could controll the known ones but only accept the presence of unknown ones - your more working test is experience here, as experience takes into account both the factors (those logically controllable and those unknown), (also extrapolation in thought (not thought, but extrapolation of it) rarely works though it is common human mind error)

* there are of course domains more predictable and those less predictable, ** mainly the more predictable ones are things that you was already doing (are deep into), not much predictable are the things you are yet not doing - but in general imo the majority of things people do are really strongly unpredictable/unexpected

** may seem that those more logical domains as hard algorithms are more predictable than those with not so logically controllable as a team work (why it is working gooda and why not etc) but in reality i think even those hard algorithms can be unexpected at it is always some way of looking on it from unexpected side that it will change your view again

(so this was small thoughts/essay on (un)predictability ;o )

I bet through some clever trickery and using player locality you could engineer a system of virtual boxes or spheres of players that overlapped and allowed you to grow or shrink on demand.

The problem you run into is the marketplace crowd. I'm in the East corner. I have a neighbor, and interact with him/her. He/she has a neighbor, and interacts with him/her. ... neighbor, and interacts with him/her, and now we're in the West corner.
The problem is not growing and shrinking with demand -- that's easy.
The problem is being able to slice the load in a way that both allows you to grow somewhat linearly-ish (say, N-log-N, or N-sqrt-N) AND gives you consistent physical simulation for all actors involved.

As you said, character/character interaction is N-squared; connections (and web architecture) is all built around scaling out the N problem. No real-time physics simulation engine exists that scales out across machines along the axis of the number of cross-interacting entities, although they exist (expensively, see DIS) for making each separate actor extremely complex.

If your needs match those of Farmville, an EC2 based web solution is great.

Right, but I think you're suggesting that games fall into either "real-time physics simulation" or "Farmville", and my MMO experience is definitely more in the middle. For the most part, the only physics being simulated is a position and velocity for each character, and there can be significant lag or even jitter in those cases without any downside to gameplay as there are no aiming mechanics or similar. With that holding true, and movement being fairly easy to progressively degrade (eg. just by sending less frequent updates), the performance constraints are significantly reduced.

The main problem I would then face, with a system that doesn't need strong guarantees on physics, is that I don't see any easy way of decomposing functionality across multiple servers without the usual proxy or ghost entities to stand in for multi-entity actions. Changing every action into a 3 or 4 stage state machine working with async messages is unwieldy, especially if you need to lock each actor while a state machine is in an intermediate state. And implementing some sort of attack bonus that is conditional on the current state of an actor on another process... possibly not practical at all.

It seems to me like you're overthinking the architectural question.

Divide things into the smallest chunks that make sense as isolated processes. If you have a lot of game logic for combat, such as your attack bonus example, you have the combat logic live as a unit - no smaller.

Yes, that's a bit hand-wavy and expects some degree of mystical intuition, but frankly if you can design good modular software you already have most of the skills to think out how to decompose problems in a distributed environment.

For the situation you're describing there Kylotan, you may be better off with a distributed STM approach rather than thinking of it as "purely async". They're not mutally exclusive.

http://avout.io/ is an example of something I'm using in development right now.

It's very similar to MPI approaches for cluster computing.

I don't see any easy way of decomposing functionality across multiple servers without the usual proxy or ghost entities to stand in for multi-entity actions.

The question is not how complex the simulation of a single unit is, but how complex the interactions between units are.

If you have a transactional store like Redis, you could run each individual unit on whatever machine strikes your fancy, and let Redis take care of the race conditions and consistency. The problem is, you are then no longer guaranteed latency, and even for loose physics MMOs, at some point, latency matters.

Also, your Redis instance will now become the unit of contention. Redis cluster doesn't support transactions that span multiple horizontal instances, and thus you end up with the same scalability concern as the dedicated-physics-server has: All entities that interact must be on the same physical host. Except, now you have the overhead of a generic database (Redis) to contend with, in addition to the physics. Keeping the data on the physics server is likely to give you a better linear constant for that, and the scale-out situation is exactly the same, so using the typical web scaling framework actually is more expensive and scales worse in this instance.

That's an issue with your specific example. Redis will have that issue, Riak , Cassanda, Dynamo won't.

You trade off some latency, but gain high consistency and horizontal scaling.