P2P persistant MMOs without any central seed server are fraught with insecurities and technical problems.

I'm also slightly confused by the apparently technically knowledgable making the (horribly mistaken assumption) that bandwidth and hardware costs are the only reason for levying a subscription, and that distributing the service is the silver bullet that will reshape the MMO market's (fairly successful) business model.

Subscriptions are primarily a mechanism for a firm to handle the support costs a persistent MMO incurs - in terms of technical support ('I have a bug!'), in-game support ('I was griefed / scammed') and long-term development (additional content) and maintenance.

It's only natural that a product or service which perpetually levies an expense on the provider has to levy a perpetual expense on the consumer.

A seed-served trusted peer mechanism does alleviate the central hosting costs, but doesn't help the support teams isolate and identify problems. A corrupt installation on a peer could lead to crashes, and or denial of service attacks using the game peer mechanism. The development time required to work against attacks as they are discovered can end up costing a not insubstantial amount when compared agains the monthly cost of having a secure network infrastructure.

Untrusted peers (a true peer-to-peer network) is not a viable infrastructure for a commercial service. The vulnerability to attack of such systems precludes their use, as does their inability to be centrally regulated or updated.

In terms of game design, persistence (across many clients) by its nature REQUIRES a known location of the persistent data. In a P2P environment it is impossible to guarantee (and dangerous in the extreme to assume) that data is available when required - consider the 'waiting for sources' stage of many P2P networks.

Ok, so persistence is out. What about linked metaverses (something I have put a great deal of thought into), or short-term, no-persistance designs for large player loads, such as full scale clan v clan FPS games. These work, but again, a central server is required to act as lobby. This could be done through GameSpy (I think I've mentioned already), and since known peers are involved, and can be tracked by a separate user name and password (to the game's actual clients, which connect through a private mechanism to one of the peers involved) this allows flexible-size games to be built. However, a known or fixed world layout, with peers responsible for a specific zone, and only that zone is impossible to implement (since peer availability is not guaranteed).

Seamless worlds are also out in a true peer-to-peer environment. The intra-server communications required (for preloading, client transfer etc.) in turn require a discrete, known spatial hierarchy. The most efficient way to negotiate hierarchical load balancing is through a master server which, monitors traffic from the servers in its spatial domain, and issues preload and spatial domain shift events. Allowing peers to negotiate amongst themselves (assuming a mechanism for peer notification is used a la gamespy) for spatial dominion is a recipe for disaster and massive lag as client bandwidth is subject to change. Randomly. Unless you can assume ALL your peers are on the backbone, you're going to get contention issues, which can lead to spurious load analysis results. So while their machines argue the toss about who should be hosting, your players will get pissed off and leave.

Of course, in a true peer-to-peer environment, this negotiation has to take place all the time as peers drop in and drop out. Bear in mind that the negotiation is only part of the headache, as a full update of the entire zone's content needs to be transferred to the new peer in order for it to pick up where the other one's left off. Load balancing 'live' like this is a lag hit, and in a peer-to-peer unmanaged environment, it WILL screw up the negotiations. Similarly fun events like a peer disconnecting mid-transfer have to be managed, meaning that the game must continue to run on its previous peer untill the update is finished. If that peer is already under such use that the domain transfer is out of date when the transfer completes (likely), all players in that domain will experience a 'deja vu' glitch. Unpleasant, and probably frustrating.

This is not to say live load balancing doesn't have its place - I have implemented it in my design, but I strongly discourage its free use. It can alleviate persistant lag, but only in situations where the transfer can be completed without a new negotiation occuring - easy enough in a regulated environment.

Nathan Baum: If there is no central game server, how does one connect to your distributed MMO? In every distributed system I'm aware of, there is, at the very least, a global server that coordinates 'client-side servers'

won't this imply having to always play on the same computer? If so, the playerbase playing from cybercafés will be reduced, since usually, you don't always play on the same computer when playing there...

Quote:

Original post by Anonymous Poster There is a very good model for persistency for a p2p mmo game. Each player should be made responsible to store their own data, but in a digitally signed form. In this case checking the data signatures against the signatures of the signing computers (server keys of p2p zone servers or the instance keys of the zones) is enough. In case of cheating, the cheating zone's signing key should be invalidated and each player must roll back to their newest valid data. (last known good state) This way each server, each zone and each player can be identified by it's signature and every data can be checked. Both server and client licenses can be revoked. Revoking a server licence will mean a rollback for those who had their data last signed by the server in question, and revoking a user license will mean a ban for that character from the world. This model doesn't solve scamming and other social issues, but stops client side cheating and makes server side cheating very hard.

Hypothetical case #1:

I and a friend (or an alternate character) log into a new zone. I give my friend all of my valuable items. The friend leaves, possessions in hand and rightfully signed by a legitimate host.

I leave, and then, since I can always revert back to a prior checkpoint, I do so, discarding the signed data received from this zone. I now have all my items as well. Photocopying made easy, so much for your economy.

Hypothetical case #2:

I have access to a rogue server that I can control, that is presently considered trusted. I take advantage of this to "instant-level" or otherwise improve my character. I then take the character to one or more other, perfectly legitimate, and well-trusted servers, allowing me to launder my bad behavior through legitimate servers, and now, even if the rogue server is detected, I'm protected.

--Dave

With case 1, you have to allow players to come back if their client dies unexpectedly and doesn't save the game state: they might have a power cut, or some other app may cause their computer to lock up. In this case, the two states (held by the client and a zone server somewhere) won't match. You said in your post at the top of this page that such situations would result in a client rollback – which leaves you open to copy hacks.

Quote:

Original post by Anonymous Poster for #1 : Your characters state is audited whenever u enter the p2p net. I would imagine there be persitance servers which check your characters last known signature vs ur current one. Obviously they won't match if u fiddle with the characters so blantantly. Every transaction which affects your character is recored and the signatures of your pre/post character states are stored. A signature in this case is a strong hash of your characters complete state. When u enter a p2p net game you have to upload your characters state to be verified against the audit logs. You can try to fiddle with the states to make the hash match but thats very unlikely, since this isn't arbitary binary data, but game data which is much more constratined. Using these strong verification schemes both server/clients can agree the states are same/good which allows both to operate upon them unhindered as long as they do it in lock step fashion. #2 : Persistance servers store transaction logs, which exist outside the domain of a trusted simulation server. Transaciton logs store who/where/hows of the transactions. If your corrupt server is detected, which is fairly easy, as your giving urself +5 super dupper platemail out of no where is going to look very suspcious to the automated auditing systems. The entire transactions of your server ( ie all items, gold, etc.. ) can be rolled back / recoupped from the entire world state. ( players who horded gold from your server would possibly have negative gold, ie they owe the world gold which would proably be recouped by the sale of their items at auction ;) ) This ofcoruse brings into bare the concept of fairness. If a person wanders into a corrupt server and gets an item, do they lose that item once the server is found to be corrupt? Mostly likely a level of fairness has to be determined for innocent bystanders. Proably a cuttoff in terms of monetary value would be used. -ddn

So, what you've done is decide that the zone servers aren't trusted, the audit servers are trusted. That doesn't preclude attacks where I control the audit server, however.

Additionally, I am assuming that you are checkpointing the transactions on a batch basis. If you aren't, if each transaction is checkpointed from the zone server to the audit servers, you have removed the entire purpose of having a zone server, AND you've made the network traffic and code complexity insane.

So, again, back to our first attack.

My buddy and I join a zone. I give him all my belongings, he exits the zone. He is checkpointed to the audit servers. There is some arbitrary length of time before I get checkpointed. If I can take the server down (or drop its network connectivity) before I get checkpointed, then we've duplicated everything. Or, as has been mentioned, I can drop my machine, so I never receive the checkpoint save information, if you support any kind of a player side rollback.

As far as our second example, I don't have to have a server that gives me the Plus 5 Sword of 0wnage. I just have to have a server that weights the odds somewhat in my favor. I can then take those buffed characters and move them onto other servers.

How far back are you planning on keeping that audit trail (lots of storage!), and how much CPU are you planning on spending auditing the game history of every player?

This, again, is an awfully large amount of work for the dubious pleasure of distributing your server farm.

--Dave

So, let me make sure I understand this.

Every zone server is going to log every transaction in real time to multiple untrusted audit servers (which themselves may or may not always be available).

Whenever you log into a zone, the zone will then back-audit everything about your character by querying the audit servers (which may or may not have been involved in any of your previous transactions) for a vote?

And somehow this is a good idea?

Quote:

Original post by Anonymous Poster [...]As I mentioned before, a code scrambling scheme (constantly mutable) could freeze out the majority of hackers (and ongoing auditing the task code with checksums/etc.. to detect 'unreliable' peers). That method would require a central secure server to implement.

Code obfuscation, when efficient at preventing people from understanding it, generally involves SIGNIFICANTLY decreasing the runtime speed of code, to such a degree that doing it on anything that was run frequently (such as the physics simulation, the combat system, etc) would make the game unplayable on anything but the highest end machines. The reason that systems like Valve's Steam download small security modules is that putting the protection in the game itself would make it very difficult to protect AND execute at a reasonable speed. However, making the protection 'external' to the important code in such a way introduces a number of vulnerabilities.
It is probably possible to create a compiler that creates fast but 'weird' code, but it still wouldn't be impossible to hack. Forcing clients to download a new version each day (or each week, etc) and having the compiler make significant use of random numbers (meaning that instead of selecting the 'best' instructions based on size or speed, it would select based on randomly assigned weights) would slow hackers down, but it would also eliminate a large base of players that are still on slow connections. Oh, and good luck writing a quality debugger for a custom compiler, and pray that you write the compiler correctly, because figuring out that it's outputting wrong instructions once in a while when it randomly selects which instructions to use is going to be a nightmare.
This is the most realistic option, though - you could write a custom back end for GCC and use that for all release builds. As long as you don't distribute the GCC executable and you don't link against the gnu libc, you should be fine keeping the source private (though I hear they're trying to fix that loophole in the GPL). If you avoid other commercial protection systems and solely rely on this, you can probably keep the executable small enough that you wouldn't lose too many customers. It might be wise to add as a post-build step a modified UPX that could use information about the random numbers used by the compiler to efficiently compress the code (without such information, the code is practically just random data and thus not very compressable).