The genre of Massively Muliplayer Online Games (or MMOGs for short) is a relatively new, but increasingly popular one for game developers. Encouraged by games such as Everquest, Star Wars Galaxies, or Ultima Online, many new developers are inspired to create designs for huge online worlds with never-before-seen levels of interactivity and detail; professional developers may be more drawn by the constant stream of revenue such projects can provide after launch, in contrast to the usual once-in-a-blue-moon payment from the publisher. The market for MMO games is particularly strong in Asia.

But while such games are popular, they are also one of the most complex projects you will ever encounter. Leaving aside the game design issues - creating and balancing a virtual society that can accommodate thousands of players - the practical issues of developing and running an MMO game are extremely challenging to navigate. This article seeks to provide an overview of some of the issues you should consider when thinking about an MMO project; to give you some of the questions you should be asking, as well as a few possible answers.

Servers, sniffers, systems and suits

The issues I will cover can be broken down roughly into four categories, though there is some overlap:

• Connection issues: All issues regarding the actual transfer of data between the client and server - security, stability, speed, protocols, and so on.
• Operating Platform issues: All issues regarding the architecture of your server platform. Things like load balancing, backups/redundancy, clustering, etc. Much of this category is common to standard Internet applications, and is not game-specific, but is still of the utmost importance.
• Technology issues: All issues regarding the architecture of the system you run on the server. This may seem similar to the previous category, but focuses more on game-specific issues, such as cheat-proofing and dynamic updates.
• Business issues: All issues regarding the business side of running an MMO game. Hobbyist developers may be tempted to skip this section, reasoning that it's not relevant to them; in fact, the costs and business details are what most frequently cause hobbyist MMO projects to fail.

One category which you may have expected to show up - client issues - is something that I do not feel I need to cover, as the issues inherent in writing a client for an MMO game are pretty similar to those for any other type of network game.

Connection Issues

In any networked game, you need a system for sending data between nodes in the network. Given that we're talking about MMO games - and all MMO games use a client/server model of communication (as opposed to peer-to-peer, which would suffer major security and management issues) - we can assume that we're talking specifically about the connection between the client and the server. What do we need to be aware of when designing this part of the system?

Security

In general, you do not want third parties to be able to screw around with the data that you're sending over the wire. Interception of packets could lead to a victim's game account details - such as the username and password they use to log into the game - being hijacked. A more worrying scenario is the interception of credit card details, in the case of a game which lets players pay their subscription fees from within the game.

Encrypting every packet you send is probably impractical, not to mention largely unnecessary - intercepting a packet which gives a player's current position in the game world may help someone cheat, but the effort required to obtain such a small piece of information makes it unlikely that people would bother with it. Still, it is likely that you would want the facility to transmit data securely, even if you do not make use of it all the time. So how will you provide that? Which pieces of information will be encrypted and which won't?

Authentication

How can you ensure that the packets you're receiving are really from your client software? How will you prevent someone from hacking together their own client software, which allows them to do things you don't want them to? How can you be sure the source IP address is "safe?"

Stability

Players will not be happy if their connection to your server is cutting off every few minutes. Whatever technology you use to connect your servers to the Internet should be very stable; protection against things like Denial of Service attacks is also something you may want to invest in. Running your MMO server on, for example, a home broadband connection which disconnects for five minutes every few hours, is not advised.

Bandwidth

The connection will be handling packets from potentially hundreds of thousands of clients; the realtime nature of most games means that each player will be sending and receiving many packets per unit of time, compared to non-realtime services like websites. You need to have a connection which, as a whole, can handle this much data; several gigabytes per hour is not an unreasonable expectation. It's unlikely that you'll be handling everything through a single pipe.

Latency

Sadly, it's not yet possible to send information to the average client through quantum entanglement; so, the time it takes for your packets to go from server to client is significant. When sending packets across the Internet, your data will be potentially handled by a pretty large range of devices, each of which running under different loads and imbued with different levels of power. The time taken by a packet to reach its destination will vary, and it may not be small. How will you attempt to minimize latency, at least within the parts of the connection you control? How will you deal with packets that take a long time to reach their destination, and packets that arrive out-of-order?

Failure response

What will you do, both at the client and server ends, if the connection drops or packets are lost? This has implications in your game design - particularly dealing with players who conveniently 'drop' from the game just as they're about to be killed violently - but it bears thinking about on the technical end as well. Should the client try and reconnect, or should it wait for the player to instruct it to do so? Should the server give the player a 'grace period' of a few seconds to reconnect, perhaps based on their ping time, or should it treat the disconnection as a proper one immediately? When it comes to the per-packet level, when will you use guaranteed delivery (which causes all lost packets to be automatically resent, but requires extra packets to be sent back to the server for every packet that is received by the client), and when will you use non-guaranteed delivery?

Protocol

What conventions will you use in your networking protocol? (By 'protocol' here, I do not mean TCP or UDP - rather, I mean the custom protocol that you build on top). An example would be strings: you could go with C-style null terminated strings, or you could go with Pascal-style strings where the length is prepended to the string. Will you stick to network byte order (big-endian) for your data?

Operating Platform Issues

Your 'operating platform' - server-side, at least - is the entire server system. It may well be spread over multiple machines, multiple processes and applications, and even multiple continents. What do you need to think about when designing the operating platform for your server?

Load

How many players do you expect a given element in the operating platform to have to handle at any one time? How much spare capacity do you want to build into the system - expecting and thus designing for 10,000 players in the beginning is fine, but if you later on want to expand your market share and thus attract new players, you're going to need to either have the capacity to handle them in the beginning, or come up with a concrete plan for extending the platform at a later point in time. (By a "concrete plan" I do mean something pretty specific; "buy another server" is not very good, while a detailed breakdown of what you will need, step-by-step instructions for integration into the platform, and how much you expect it to increase capacity by, is better). Also, be aware of the difference between total players and concurrent players - even if you have 25,000 total subscribers to your game, you may only have 3,000 players online at any one time. That may not affect the requirements for some parts of the system, but it can heavily affect others.

Distribution

How are you going to spread the load over multiple servers? Will you have one server per subsystem, or will all servers run all subsystems for some part of the game? (For example: will you have one server dedicated to physics processing and another dedicated to AI, or will both servers run physics and AI for separate parts of the world)? How will load distribution be controlled - will you use a single, central 'depot' server, or will individual servers 'pass off' excess work to each other?

Security

You have to provide a way for the outside world to connect to your servers; otherwise, none of your clients could ever log in. But you must be careful; every open port is another potential avenue of attack on your platform. Take the utmost caution when dealing with data from the client; your aim is to make it impossible for someone to crash your servers by sending in a carefully crafted packet. What measures will you take to verify the integrity of the data you're getting, or to protect routines that work with that data? Also, you need to protect your internal services - things like database servers - by keeping them on a network completely separated from the Internet; how will you architect your network to allow the necessary machines to communicate with them, without providing a route to them from the outside world?

Redundancy

Say you have a hardware failure, a power cut, or your security fails and someone manages to crash your servers. How will you go about restoring the world to a state as close as possible to before the problem? You can go for backups; dumping the entire world database to a tape drive every day may be enough for your purposes (and besides, there may be reasons beyond technology failures for keeping backups - such as responding to endemic cheating), but how precisely will that be achieved? Which parts of the system will be backed up?

Another thing to consider might be something like RAID - it means a higher initial cost (instead of buying one 120GB disk for the server, you're buying two or three plus a RAID controller) but it means you've got a second layer of defence before you have to go to your backups (if one disk fails, you can respond in a controlled way to replace it while the game continues to run - though you don't want to hang around, because the other disks could go too). If you do go with a RAID array, how many disks should you invest in? Which RAID level do you want? What's your plan of action when the RAID controller reports that one of the disks has failed (given that you usually can't hot-swap disks around)? How will you cope if the entire RAID array fails (the controller blows out)? Even if you do include a RAID system, it will only account for what's on disk and not what's in memory; so how will you restore what was in memory? If you're not even going to try, how will you minimize the data being kept in memory at any given time, and ensure that it gets written to the disk as often as possible without slowing down?

Synchronization

When you've got a system composed of multiple processes communicating with each other, you're bound to run into synchronization issues some time or another. It's particularly a problem for the database servers - what if two machines are accessing the same record at the same time? Do you implement a 'locking' system so that the first one there gets it and the others have to wait or drop out with an error? Perhaps you go for atomic operations? Or just allow it to happen and to hell with the consequences? The last approach might be perfectly valid if you don't think the chances of a given record being used by multiple other processes are high.

Technology issues

In this section I'll discuss some of the MMOG-specific technology issues, covering both client and server applications (the "server application" being the collection of programs you run on the server-side operating platform).

Caching

Caching is the practice of storing resources that you're more likely to need in a place that's faster to get to than the regular data storage. In MMO gaming, the client programs will pretty much always be caching data - that's because the "regular data storage" is all the way over the network at the server, so accessing it is likely to be very slow. So, we're talking about data that the client will need for the very next frame - models, textures, sounds, and other assets for the player's current status and location would be an example. A key concept with caching is "temporal coherence" - the idea that something is the same from one frame to the next. If something is temporally coherent, there's no point re-loading the data from the server, so you cache it. Beyond that, though, what do you want to cache? How much of that content needs to be picked up dynamically - from the servers - and how much can be included in the client software (i.e. on the CD)? The rule of thumb is: cache anything you might need in a hurry, especially if you can't predict when you'll need it. Graphics for an in-game menu would be an example - you have no way of knowing when the player will bring up the in-game menu, and you definitely don't have time to sit around downloading the graphic from the server. Given that they're unlikely to change, they'd be prime candidates for including in the client software.

At the server end, you'll probably find advantages to caching things as well. You might want to cache things like level geometry around spawn points, so that it can be sent more quickly to players joining the game, rather than having to pull it out of hard storage; or maybe you'll cache information about a specific item that people keep asking for. The best way to find out what to cache is to monitor the requests and see which ones come up the most frequently; a truly rock-solid caching system will do exactly that, and has the advantage that it can adjust over time (i.e. if people hardly ever use a given spawn point, it can detect that and stop caching it).

You'll also want individual servers to cache information within the operating platform. Say you're processing collision detection for a bunch of entities; you'll get a good performance boost from fetching them all into the cache and testing them there, rather than fetching them one-by-one, testing, and then dropping them so that you have to get them from the database again next time.

Also, at both ends of the wire, How large will caches be? Client-side, you'll probably be limited by both the amount of system memory and the amount of hard-disk space you can reasonably use; server-side, though, you've got more control. You should definitely take cache sizes into account when designing your operating platform.

Dynamic Updates

The ability to dynamically update the game - adding new content, removing existing content, or changing content - is crucial. New content keeps the game from getting boring and thus keeps the players playing; there may also be bugs or gameplay balance issues which haven't emerged until several thousand players are present in the game (the latter is especially common). What's your procedure for updating the server application? How about the clients? Will you just stop all the servers completely, make the changes, and relaunch it? Perhaps you'll take down servers one by one - after moving their load off to other machines - update them, and then bring them back into an operational state? That means that until you've done all the machines, some parts of your network will be running the 'old' version and some the 'new' - how will you deal with that? Or perhaps you'll avoid bringing the machines down at all, providing a mechanism for the server application to 'assimilate' changes while it's still running?

The effects of content updates can be extremely extensive. Say you want to create a brand new building in a specific place - will you just sit there and wait until the space you want becomes clear of players? If you're placing the building in a town, that could be quite a wait. So perhaps you'd go for some system which lets you 'mark' the space as about-to-be-built-upon, turning it into a no-entry zone for players until you're done building? The addition of a brand new chunk of geometry to the world would force players in the vicinity to download it from the server into their caches - how would you give the clients 'advance warning' so that they could pick up the mesh before it's needed, to prevent an awkward pause when you place it?

Then, say you want to remove that building a few months later. What do you do with players (or NPCs, for that matter) who are currently inside it? Just drop them on the floor outside with no warning? Would you tell clients that the assets for the building no longer need to be cached, and if so, how?

The most difficult type of update is usually a code update - where you're changing the actual software running either client-side or server-side. You may well want players to refrain from playing until they've installed the client-side update (for example, if it closes an exploit that some players were using to cheat with) - how will you enforce that? Will it be done from within the game, or will players have to exit and visit your game's website to get the update?

Security

It's probably been said that computer security is all about being openly paranoid about your users; if it hasn't, then I'm saying it, because it's true. Any user is a potential cheater, cracker, or malicious script kiddie aiming to take down your servers or generally ruin the game for other people. As such, you should never trust anything information coming from the user - even if you have authentication measures in place to try and ensure that data from the user really is from your client app. Say that your client app is repeatedly sending the position of the player to the server, and the server is simply copying this position into the database as holy writ. An enterprising cracker could modify the client application to send a position value of his choice; and, presto, he's granted himself the ability to 'teleport.' Even if your authentication system prevents modifications to the client software, there are still ways of modifying packets on the fly. If someone finds a way around your authentication, then it's as good as non-existent. So look at things from that point of view, of the position of having no client authentication.

The most secure situation would be to send data to the server purely as a stream of input commands, and have the server do the actual processing of those commands to move the player around and so forth. Even then a cheater could still take advantage of you; he could record a stream of keypresses and send them all to the server in quick succession, allowing him to "press keys" faster than a real human actually could. So you'd still need some kind of 'sanity checker' to watch the user input streams, just to check that what it's seeing is reasonable input for a human player. Of course, this whole approach isn't used because of the processing power it requires; you'd effectively be moving the load from all of your clients (who may number in the hundreds of thousands) onto the server, which is a pretty significant load. The MMO client-server setup represents a pretty significant distributed computing platform, but you have to be careful about what the clients are actually doing.

As such, it never hurts to keep things fairly tightly locked down on the client side; server-side security systems are all very well, but it's nicer if you don't have to use them. Pirating an MMOG client is usually pointless (you still need to pay a subscription fee) so copy-protection schemes are generally unnecessary; instead, the client should be watching out for modified data files (perhaps through a checksumming mechanism), unsavoury-looking programs in memory (particularly so-called 'trainers' that attempt to alter the client's memory while it's running), and debuggers. All three could be signs of someone trying to cheat the game.

Synchronization

Here it is again, though in more of a gaming context - you need to keep all your clients, and all your servers, synchronized (or as close to it as possible). That doesn't just mean that they need to all display the same game clock time; they all need to be working with the same set of data. A player shooting at an enemy will be less than impressed if that enemy actually moved away several minutes ago and they're shooting a ghost (but couldn't tell because the server isn't keeping them up to date).

This issue is particularly prominent in MMO games where different servers are handling different regions of the world. When the player travels from one region to the other, and hits a 'server boundary' (i.e. reaches the edge of the area controlled by the current server), he must transfer to a different server. If the two servers are out of sync, it could cause major problems; for example, standing in one server the player sees a tank rolling in his direction but still far-off; when he switches servers, he discovers that the tank is actually right on top of him. Even to an observer it can look very strange - an entity can be seen to 'skip' boundaries as an out-of-sync position is suddenly updated to a correct one.

Business issues

By now I've shown you that MMO projects are pretty complex beasts to plan; combine the work required to create a stable server application with the cost of the hardware for a strong operating platform, and it adds up to a simple truth: MMO games are expensive. Even with that in mind, you may still have the resources and/or backing to go ahead with the project; in which case, I salute you, and present the following issues for your consideration.

Business structure

Once you've developed the MMO game, are you going to run it yourself, or are you going to set up a separate company to do that? It's like a franchise; the new 'firm' is purely dedicated to running the game that you have developed. It's not hard to find ways to flow money from the new firm back into the old - royalty payments would be the obvious choice - and control can be kept by simply owning the new firm. The advantage is that the finances of the game and of the developer are kept separate; if the game is making losses, it doesn't eat into the developer's own finances, only the game runner's.

Maintenance costs

It won't surprise you to know that if you're going to run a load of servers, you need to supply them with electricity, keep them in a building somewhere, and probably keep that building secure. You'll probably also need to employ a few people to keep them running and deal with crashes and crisis; of course, that means you'll need to heat the building, too (though the heat from 30 servers may be enough, if you can persuade your staff to huddle together a bit). So purely to keep the game running in the same state as it was at launch, you're already running a small office. If you're lucky, your publisher will handle things like subscription payments for you, but you should consider the scenario in which you have to handle that sort of thing yourself. You'd try and automate the actual payment process as much as possible, hopefully to the extent that a problem-free signup requires no human interaction at all; but you'll still need a staff of people on support, to deal with the more severe payment and connection problems.

Content development

As noted before, games which aren't continually revised and updated tend to lose player interest. So how will you go about developing new content for the game? You could keep the team who wrote it in the first place as the people behind the project, but that does keep them from moving on to other projects. You could subcontract. You could hire a permanent development team to keep adding to the game; if you've opted for the 'franchise' approach I suggested earlier, this may be the most attractive option - hire a game designer and crew of artists and programmers to "grow" the game from its initial launch state.

One option which doesn't seem to be commonly used is that of using the player base for content. It's quite possible that there are talented artists or designers in the community; perhaps it would be worth setting up a mechanism whereby they can contribute content they'd like to see in the game?

Revenue stream

It's something of a foregone conclusion that you will want to generate a constant stream of revenue from the game somehow - at the very least to pay for maintenance costs, assuming no new content is being developed. How exactly will you do it? Purely through subscription fees? Will you charge for the initial client portion of the game (the retailed, boxed copy)? How will the subscriptions work - will it be one-fee-fits-all, or different subscription levels for people playing the game to different levels? Will there be 'unlockable content' for players paying more?

There are a few other options to consider. One of the biggest is advertising - is it plausable to try and sell product placements in the game? Unreal Tournament 2004 did this with a fair amount of success. The fantasy setting of many MMO games often makes product placement difficult, but it's still something to be considered.

Middleware

There's already middleware out there aimed specifically at MMO games - Butterfly.net's Butterfly Grid is one example. Such middleware aims to provide both an operating platform and server application framework, upon which you can build your own game without needing to deal with many of the issues raised in this article. There is a certain loss of control, and of course it isn't free, but it's another option that should definitely be considered.

Conclusion

This article has just been a taster of some of the issues facing a developer who wants to enter a project into the MMO market. I've more-or-less sidestepped the game design issues inherent in such games - and yes, there are many further decisions to be made in that area. It is quite telling to note that of all the MMO games out there, only eight or so have more than 50,000 subscribers. (http://pw1.netcom.com/~sirbruce/Subscriptions.html)

Still. While it may be tough work, the MMO genre does provide some very unique opportunities for game designs, most of which have not yet been explored. As the proliferation of broadband technologies continues across the world, the size of the target market increases, and MMO gaming looks like it's going to stick around for quite a while.