Hi,

I am in the middle of the process of designing browser-based MMORPG architecture and I am trying to define a generic set of functionalities which these kind of games typically needs. Hence, the challenge here is that these features or aspects should not be defined specific to some game - they should be applicable across typical MMORPGs. The games I am referring to are the likes of World of Warcraft and Silkroad Online, etc. Thus, I am very well aware that task I am doing is big -- but I am not trying to implement MMORPG from scratch here, I am simply trying to come up with an architecture proposal which is larger in content than many of the previously written articles, as they typically describes only the communication between the game client and the server and skips the issues raised around authentication / security and alike.

That said, I already went through the material regarding previously designed architectures and I conclude that non-functional requirement of "scalability" seems to be the deciding factor in MMORPG development.. and the latency, bandwidth and cpu usage basically makes developing these games remarkably hard and affects the scalability aspect of the game. However, there are also other things which I identified, and the initial list is here below:

Functional requirements:

• Authentication
• Registration
• Character creation/loading
• Game loading/saving
• Game state calculation/distribution
• Chat
• Quests
• Fighting
• Movement
• NPCs

Non-functional requirements (high-level overview):

• Security
• Reliability
• Scalability
• Availability
• Traceability
• Maintainability
• Usability

The way I am planning to use this knowledge is to find things which should be documented within the architecture and to be part of the architecture... the evaluation of the architecture is another thing -- for instance, it is very unlikely that I implement an MMORPG to test that it satisfies _all_ the requirements, I simply don't have time to do that. But I will come up a way of evaluation things once I get there.

But now, any comments will help, thank you!

Edit:

Realized that my "functional requirements" are more like "features" but nevertheless I think the goal of this post is still clear. I am planning to format them properly withing the actual work.. same applies for strict limits of non-functional requirements.

First, there is a big caveat here: On the continuum between "extremely general," and "game specific," it's really hard to find a viable, useful, middle ground.
All MMORPGs need servers, so general server-management infrastructure is probably useful -- think "docker" or "elastic beanstalk" or such. Totally general, not game specific.
Similarly for storage infrastructure -- pick something general and well known, either relational (MySQL or Postgres) or NoSQL (Redis or Cassandra, or the cloud-based BigTable or SimpleDB.)
All MMORPGs need some kind of graphics, and user input, and audio. There are APIs in browsers for these things, and those APIs have a bunch of warts, so compatibility helpers are useful, as are 2D and 3D renderers, as ar even full-fledged game engines. That's a super-big, hundreds-of-man-years area.
Regarding quests, that's also kind-of generic, but also kind-of genre specific. Quests are found in most games -- campaign mode of RTS-es, single player FPS, single-player RPG, action-adventure, and a bunch of other game kinds need this just as much as "MMO" anything.

Now, once you start getting into any more details, you end up with genre specific requirements.
There are services around the periphery that can be useful to all modes (game matchmaking, server selection, account management, etc) but the gameplay does dictate the optimal architecture, and if you don't choose the optimal architecture, you will be at a cost disadvantage compared to your competitors.
Do you need server-side simulation? Is that simulation turn based or real time? Does it include physics simulation or just game rules? The approach you take matters. Travian is different from World of Warcraft is different from Planetside.
Calculate-forward-on-request (like Travian, etc) is very different from ticksd-simulation with physics (like Planetside.) Even the way that you represent state and marshal to/from network data may be different, based on requirements.

Hi,

Thanks for the good points - as always - your post actually gave me some more things to consider. :)

And you are right about difficulty of finding the middle ground, I feel dizzy every time I have to think about it.. but I am willing to give it my best shot; I am extremely happy if I come up with a solution which seems to have all common pieces in place and doesn't go too deep regarding the details.

That said, your comments made me think that I should emphasize that I am designing architecture for browser-based real-time MMORPGs and type of the world it should support is a mirrored world approach. I have read that continuous world and server-to-server communications requires very different things from the underlying architecture.. and yes, I have had plans of using Postgres for account information etc., plus MongoDB for the game data.

Nevertheless, I think I am starting to see the big picture now, but because designing architecture and validating it is a whole different thing, I guess I have some problems further down the road.. but that is already a topic of its own and I will post it later if I see it necessary. But thanks!

plus MongoDB for the game data

Where I come from, friends don't let friends use Mongo. You should look really hard at alternatives, such as Cassandra, or Redis, or SimpleDB, before you actually go down that route.

However when you say "game data," do you mean things like "current hitpoints" and "location of NPC X"?
Generally, putting that kind of data into any kind of network-attached memory is too expensive.
Game servers will keep that kind of data in-RAM, and make sure that clients are connected/routed to the appropriate server instance for the level/area/zone they are currently observing/interacting with.
When something important happens (pick up loot, etc) the game server will fire off an update message to a back-end database; other than that, if the game server crashes, players get kicked off and objects reset to initial locations.
Given that well-debugged servers crash very seldom, this is usually the right trade-off, because trying to persist ephemeral data has an unreasonably high cost.

plus MongoDB for the game data

Where I come from, friends don't let friends use Mongo. You should look really hard at alternatives, such as Cassandra, or Redis, or SimpleDB, before you actually go down that route.

hehe.. well, I have used MongoDB before for game data (only a demo game) and for data regarding a web service (only 700 - 900 users online at once but about 200K registered, if I recall correctly) - maybe I haven't spend too much time with it as I didn't find any good reasons to not to use it.

Anyway, I guess there is not much to store when it comes to game data .. things which I had planned to store to mongo are the location of a player (x, y) and items he posses (backpack and bank), experience, level and alike. Basically only the relevant info which he needs when he logs in to the server.. then after that a lot of the data (like location of NPC) will be in-RAM (as you said).

I had planned that game saving will be done in patches, meaning that I will pick up the relevant game state information (regarding players) from ram using a fixed interval and save it to database. This way I am not trying to save the game on every tick as I suspect that the database load will increase dramatically.. but if I do it like this only, I guess in the case of server crash players would lose the items they have received from rare bosses too.. doesn't sound awesome, but it would be same for everyone -- sounds fair?

Nevertheless, are you saying that replacing MongoDB with Redis is completely fine? and I can load the player state from Redis the same way I load it from the MongoDB when the client connects to the server? I haven't used Redis before, but I thought that it is not a persisted storage for data.

I don't understand the actual question. When you say "define a generic set of functionalities", what do you expect, beyond the list in your initial post? Even that is too specific (not every game has fighting, chat, or quests).

If we're being strict, all you need is:

a) A server program that can broadcast states to a 'massive' number of clients;

b) A client program that can display states and request state changes from a server.

Anything else is application-specific.

I don't understand the actual question. When you say "define a generic set of functionalities", what do you expect, beyond the list in your initial post? Even that is too specific (not every game has fighting, chat, or quests).

If we're being strict, all you need is:

a) A server program that can broadcast states to a 'massive' number of clients;

b) A client program that can display states and request state changes from a server.

Anything else is application-specific.

I think a lot of the RPGs shares same set of functionalities - those are the ones which makes them "RPG".. if you would compare World Of Warcraft, Silkroad and alike, you can see that those all has an ability to create your own character and explore magical world through quests, combat and collaboration with fellow players, there is NPCs, shops, items, leveling.. etc. I am not saying that every game fits - or should fit - this model, I just want to set some common features which should be taken into account within the design of architecture. After that, I document that these are the type of RPGs the architecture is intended to support. But I agree, it gets easily too application-specific if I am not careful and the level of detail I finally take into account is a problem of my own :P For instance, quests like you mentioned might be something I should not care that much or "evaluate" because they are only a typical feature within the game. On the other hand, implementing of authentication, registration and chat might need different services from the architecture.. and they are important enough to spend time to document.

In summary, I posted this question to find out whether I am missing something huge and get feedback :).. that said, if I would design architecture based on your a) and b) it already lacks information I would like to document.. (registration, authentication).. and btw, those are the very reasons I am doing the current work. I think 90% of the previous research states that "authentication is outside the scope of this paper" and only focuses to the communication between the game client and the game server. But we all know that proper MMORPG needs more services than that to function - and that is the information I would like to bring into the mix. And the reason I am not doing architecture for just all MMOs is that MMOFPS and MMORPG would typically have very different kind of latency requirements which means the fight between using of TCP or UDP.

I'm still not really understanding what the purpose of this is. If you have a specific game design, your RPG requirements will emerge from that. If you don't have a specific game design, then trying to reason about RPGs in the abstract sense is probably a waste of time (and has little to no effect on the architecture anyway).

Going through your list:

• Authentication - yes, this is important. But it's not really game or MMO-specific.
• Registration - this is rolled in with authentication, really.
• Character creation/loading - characters are just data, the process is just UI. It doesn't merit special architectural treatment, although you could offload this to a separate process if you felt like it.
• Game loading/saving - this is usually called 'persistence'. It's handled very differently by each game - no standard approach (nor should there be).
• Game state calculation/distribution - these are 2 separate topics, although how you store state can affect how you broadcast it.

The rest are mostly irrelevant to the 'massively multiplayer' aspect, except from a designer's perspective.

I've worked on 3 different MMO projects - 2 using in-house engines, 1 written myself (which was never tested at massive scale) - and although there was commonality across the 3, it was never something that could have been abstracted out and shared. One of the games didn't have quests. One had a persistent world, the others just persisted characters. 2 had full auth systems, the 3rd was more like IRC. All 3 had different ways of requesting state changes and transmitting them. All 3 had different ways of handling area of interest calculations. They all handled zone transitions differently. So at the high level of box-ticking they all did similar things but practically they were all architected differently.

If you have data that you want to persist across sessions, there's really no reason to not put it in a SQL database.
Worst case, "id + JSON-as-TEXT" works as a key/value store. Adding metadata (like "what zone did the player last checkpoint in" or "what level has the player achieved" or "what account ID is this data associated with") to indexed fields makes certain kinds of gameplay management easier.

Redis is great for data that needs the data structures it provides, and data that you can always put a finite lifetime on.
Data without an expiration time in Redis is bad.
Also, data sets that need to be transactional and grow are not great matches, because once you have a 512 GB Redis server, Linux will take 5-10 seconds just to fork() each time you want to checkpoint, and while doing that, the entire machine is unavailable.

The reason I recommend against Mongo is that they still haven't really solved some important long-term operational issues, such as consistent backups while keeping availability.
The recommendation is literally "run a replica in another data center and hope nobody finds a way to inject bad delete commands."
(Although their paid, hosted, trust-them-with-all-your-backup-data solutions claim to solve this -- I have no data points on how well that works or what it costs.)

at the high level of box-ticking they all did similar things but practically they were all architected differently

Kylotan, I think you're adding great examples of the main caveat that I wanted to convey.
I'd like to push back a little bit against this, though. Yes, you DID architect them differently, but did you HAVE to?
I've worked on games and projects with several different graphics/rendering and physics engines, and they are all architected differently.
But, looking at them, and squinting a bit -- they could probably all have saved time by starting with something like Unreal Engine 4, had that existed at the time. (The available offerings were less complete at the times, and thus different decisions were made at the times.)

I used to love MongoDB for rapid iteration of unstructured game data,

Then I realised that I could use PostgreSQL with a JSON column. This gives you 95% of the functionality with 200% of the reliability.

I'd like to push back a little bit against this, though. Yes, you DID architect them differently, but did you HAVE to?

It is a good point. The answer is, "maybe". :) For example, the 3 state changes approaches each had a different justification. I was able to get away with simple entity change broadcasts in my code because they were infrequent enough and because there was only one authoritative server owning an entity. The first MMO I worked on had explicitly disabled that because it was profiling as too expensive, and because it didn't play nicely with their load-balancing. The 3rd case preferred explicit notification of changes and didn't want to perform any sort of automatic broadcasting, but I didn't play a part in that decision. I think a single system could technically work in all 3 of those cases but I don't know whether it would be possible to stick within all the constraints and 'soft' requirements that way.

I'm reminded of a conversation I had with someone from Cryptic about why they felt the need to write an entirely new DB system for their MMO. Essentially their vision for the game had strong implications for the persistence mechanism, because they felt they had to solve problems that the rest of the industry didn't really consider to be a problem in the first place. I still think they were wrong, but they had a design that called for implementing persistence in quite a different way to everybody else. It ticked the persistence box, but only with a custom-made DB.