A peer to peer game where everyone runs the same simulation does not suffer from this. What you have is a client-server game, where one of the players controls the server.

That's easy to say, but all you've done is rephrase the problem. How exactly do you ensure that everyone is running the same simulation? What's stopping a group of people from hacking the client and using that instead?

Right. You're assuming the cost of sending commands is the expensive thing, and you're reducing that. But in my experience, at this scale, commands and bandwidth are not the issue. The expensive part is the logic of the simulation.

I've analyzed it myself and I can confirm that the limiting factor is in fact the bandwidth. I can provide you with more details if you really want, but the short story is that It has to do with the fact that my game is computationally very light-weight.

I'm not an expert on this in any way so consider my reply just as feedback.

What I don't like about peer to peer setups is that when the host decides to suddenly exit for whatever real life reason, the game just poofs for every other player in the middle of what they are doing at that moment. If I as a gamer look for a multiplayer game to buy and realize the only option it offers is a peer to peer setup then I would not choose this game. Cheating would be a concern too because I know how inventive people can be to exploit certain weaknesses if it gives them an advantage.

I know you mentioned your reason why you choose peer to peer. I'm just thinking that in the end if you want to sell your game it's good to think that there are so many games out there using a client - server model that players can choose from.

You make a very good point. However I actually plan on giving out this game for free. If it catches on to people, then I'll consider buying dedicated servers, although I doubt it. This is more of just a side project that I'm working on for fun; not to make money. Anyways thanks for the feedback!

Some games support host migration, which mitigates the "game poofing" problem.

^This

A peer to peer game where everyone runs the same simulation does not suffer from this. What you have is a client-server game, where one of the players controls the server.

That's easy to say, but all you've done is rephrase the problem. How exactly do you ensure that everyone is running the same simulation? What's stopping a group of people from hacking the client and using that instead?

If a group of people in a peer-to-peer game hack the client and play together, they're playing a different game to the one you made. So that's a separate issue.

If only some people in such a group hack the client, then this gets detected by discrepancies in the simulation. RTS games traditionally work in this 'lockstep' way and they will kick out clients that appear to have 'desynced'.

eg. http://www.gamasutra.com/view/news/126022/Opinion_Synchronous_RTS_Engines_And_A_Tale_of_Desyncs.php

I've analyzed it myself and I can confirm that the limiting factor is in fact the bandwidth. I can provide you with more details if you really want, but the short story is that It has to do with the fact that my game is computationally very light-weight.

Fair enough then. You have to do what is right for your game. However it's usually much easier to optimise bandwidth use than to optimize AI, physics, and other game logic. And it's probably easier and more maintainable to do that than to create and use the tree topology (which is why it's not generally adopted). But it's your game.

All other issues (mostly extra latency and reliability) aside... have you thought thoroughly whether you want to have one player know his competiting players' IP addresses? I'm inclined to say: No, you don't want that to happen.

What prevents me from holding back messages for a few dozen milliseconds (just long enough so no NACKs are triggered) and to watch on chat who is typing "lagggggg..." so I have a good chance of knowing exactly which avatar belongs to which IP address? You can encrypt all packets so I don't know what's inside, but now I still know who the packets are for.

What prevents me from selectively dropping them whenever I'm engaged in a PvP challenge with these? What prevents me from SSHing into a server located at university and running a bandwidth tester on that IP address?

What prevents me from randomly flipping a bit in critical situation? Sure, the checksum will detect them, and the client will NACK and trigger a resend (if I am so kind to deliver it, maybe I'll stall the packet again for an extra 5-10ms) but it will add at least one full RTT of delay in the most shitty situation imaginable, probably getting the other guy killed. And it's entirely innocent and unsuspicious. You probably can't even find evidence that I am cheating -- bit errors happen, eh?

And then, there's people playing via WiFi in an internet cafe. Or worse, via LTE. Say that in one sentence with "peer to peer" and "realtime". I don't know what your LTE infrastructure looks like, but I can tell you what Telekom's looks like over here. Mighty fine bandwidth, no doubt, but latency is... holy shit. Like 120+ ms to hosts in the same city.

If a group of people in a peer-to-peer game hack the client and play together, they're playing a different game to the one you made. So that's a separate issue.

If only some people in such a group hack the client, then this gets detected by discrepancies in the simulation. RTS games traditionally work in this 'lockstep' way and they will kick out clients that appear to have 'desynced'.

eg. http://www.gamasutra.com/view/news/126022/Opinion_Synchronous_RTS_Engines_And_A_Tale_of_Desyncs.ph

The only reason I initially incorporated a host into the system was to solve the whole desync issues. How it works is that players can send commands to the host relative to their current position (e.g. I want to move-left), and then the host relays the information to other players relative to the entire game state (e.g. player1 moved to x,y). That way every player is playing based on the host's game state. It wasn't until later that I realized it would be perfect for preventing hackers from messing up the physics (they can't just teleport across the map). Thanks for showing me that article also.

Fair enough then. You have to do what is right for your game. However it's usually much easier to optimise bandwidth use than to optimize AI, physics, and other game logic. And it's probably easier and more maintainable to do that than to create and use the tree topology (which is why it's not generally adopted). But it's your game.

I see what you're saying, and you're not the only person who's told me this. It's just that it works differently in my game. Thanks for the input though.

Hello samoth,

The only way to prevent these sort of situations is to join games where the hosts can be trusted (both primary and secondary). Since we're assuming that these hosts are "trusted", we can further assume that they won't be doing anything nefarious with the regular users. Of course there's no way to be sure about the secondary hosts, so the primary host would have to be on the lookout for any suspicious behavior (i.e. people complaining in chat) and take action as needed.

Choosing the secondary hosts is actually easier than it seems since the total number of secondary hosts required is = quotient((n + 7), 9)
where n is the current number of players in the game and 1 <= n <= 73
e.g. if there are 18 players, the game would require 2 secondary hosts (plus 1 primary host and 15 regular users)
e.g. if there are 73 players, the game would require 8 secondary hosts (plus 1 primary host and 64 regular users)

Another way to think of it is that you start with one primary host, and each secondary host you add (up to the maximum of 8) gives an additional 8 regular players slots.

As for regular users knowing the IPs of other players, we can rule this one out since the only IPs they can see is their current secondary host and the primary host. This doesn't pose much of a problem for DOSing since if they DOS these IPs, they lose connection as well. It's the same as a client trying to DOS a game server, no one benefits from it.

A Sybil Attack is still very possible, but you can prevent these sort of attacks by having the primary host assign a rotation to the secondary hosts based on the time-stamp in the digitally signed message that the primary host sends out.

For example in one message, a secondary-host might be assigned children: user1, user2, user3, user4. However in the next message it might be assigned children: user5, user6, user7, user8. This rapid rotation will prevent a user from DOSing any particular individual. And due to the randomness in the rate at which UDP packets are being sent, there is no way for a user to predict which secondary-host is associated with which user at any given time. This would mean though that the secondary hosts have access to all the user's IP addresses. As a result, this only works if the secondary hosts can be trusted. Furthermore, freeloading is hard to detect but shouldn't be a problem if the secondary hosts can be trusted.

One change that would result from this (not security related) is that the host would need to keep track of users who idle (no message received after a long time) in order to detect unexpected disconnections.

So in the end, it really comes down to the primary and secondary hosts to ensure that everyone is playing within the rules. Primary hosts have access to user's IPs and can change the game anyway they want. Secondary hosts have access to user's IPs and can freeload. If people are really that afraid of cheaters, they can always host their own games as well. As for people people playing on LTE, I really don't think they should be doing that haha.

Thanks for pointing out these security flaws!
achew

Visual of how the rotation works (Mirror1):

That hypnotic animated picture is awesome! :)

Randomly reassigning end-nodes to the secondary hosts will make matching IP addresses with avatars harder, but not impossible. Additionally, as a secondary host, I now do not only know some other users' addresses, but all of them. I can time round trip times to get a rough idea of where they are, too (... or just look in a geo-IP database).

So there is that guy from Denver who always beats me in PvP and boasts about it on the forums. Look, this IP address is located in Denver. Let's see how well he fares with a bit error in every 8th packet received...

But more importantly, you will face some additional hurdles doing so. Those are hurles you create, it doesn't even need a malicious user.

First, this means that you must geolocate the secondary nodes and rotate end nodes only between secondary notes which are close to them. Otherwise, you get unpredictably changing round trip times all the time. 15ms now, and 150ms the next instant because you're rotated to a secondary node in Moscow. That doesn't go well.

In a "static" topology, an end node may be unlucky with its peer, but it has the possibility of saying "this doesn't work, my ping sucks", and be assigned to another (static) peer which is closer or has a more favourable route, or whatever. It's not 100% foolproof, but it works. If secondary peers rotate, this gets a lot more complicated.

Second, there's that nuisance called stateful firewall. Which is what you find in practically every home user's router. You have to do a little dance for inbound datagrams to make it through stateful firewalls. With the same static peers, that's easy, you punch through once and be done. The firewall assumes a "connection" (where there really isn't one) and lets the traffic through. As long as data keeps coming in regularly, all is well.

With constantly rotating peers, you are in a bit of home-made trouble here. Not only are there a lot more peers, so you gotta do a lot more punching. But more importantly, chances are that if a rotation takes long enough, the stateful firewall will time out your "connection". How do you know? Well, you don't, other than packets are suddenly being silently dropped.

Or maybe the stateful firewall will only remember up to a hard total number (say, 5) of connections. If you have 8 rotating secondary nodes and a router which remembers 5, this fails.
Granted, 5 is unrealistically low, let's say 50 connections, then it fails as soon as you have 51 rotating peers. 51 secondary peers is only 408 concurrent players, so that is not very remote. Mind you, we are talking "massive multiplayer", which is more like 40,000. That would be 5,000 rotating secondary notes. I'm pretty sure the majority of stateful firewalls on home user routers will show you the middle finger on that one.

Thanks!

In addition to what you said, regular users can DOS any secondary host without reprimand since the game would still continue on for everyone else. I didn't realize this until afterwards. So although the rotation prevents regular users from DOSing other regular users, it does not prevent regular users from DOSing secondary hosts.

So instead, I've come up with yet another change which I think addresses all these problems. As for the rotation thing, well it doesn't work as I had planned, so just forget that whole thing :(

There are 2 requirements:

• The use of heartbeats to detect disconnections
• The primary host must manually choose the secondary hosts
• All primary and secondary hosts must be trusted

Also, disconnections get treated differently depending on what "group" the user is in (see table below)


With the heartbeat method, each host would have to sustain a constant amount of bandwidth. However this won't pose much of a problem since this amount increases linearly with the current number of users as opposed to a pure client-server model (see Client-Server vs. Octree).
The code is also easy to implement since all you have to do is have the client use a loop to store user input into a buffer which then gets processed and sent out as a command. If the buffer is empty, the client sends a heartbeat

To prevent false detections with the heartbeats, regular users can have a more forgiving threshold than secondary hosts since the main concern is to prevent secondary hosts from getting DOSed. Having a tighter threshold also gives a better quality control for choosing hosts with good bandwidth.

Advantages:

• Only way to get someone's IP is to become primary or secondary host
• Sybil Attack won't work since regular users are only exposed to the IPs of primary and secondary hosts, and DOSing these IPS would pause the entire game
  • Same as a client DOSing a game server in a pure client-server model; noone benefits from it

Disadvantages:

• All hosts (primary and secondary) must be trusted in order for this to work, and finding these trusted hosts is a non-trivial task (biggest downfall)
• Regular users must rely on the trust of hosters (this carries over in all peer-to-peer games)
• If the hosts are using bad internet, then the entire game is laggy for everyone else (this carries over in all peer-to-peer games)

Another thing I realized is that ALL users need authentication (not just the primary host), to prevent spoofed command/heartbeat packets

Also, I'm using the word "massive" in the context of peer-to-peer games, since I have yet to come across a game that supports more than 50 concurrent players. Not to mention, I have high doubts that my game will be that popular (although I'd love to be proven wrong :)).

This is getting ever more complex.

In all cases you're best to "keep it simple" :)

Have you looked into data distribution services yet? They've already solved a lot of these problems for you...

Yeah, although we're kinda back where we started now.

I looked into some data distribution services (DDS) and based off of my understanding, it's mainly used for users who want to distribute data (a.k.a. publishing) to a select group of people who are interested in a particular topic (a.k.a. subscribers). However if you apply this to a game, the publisher would be everyone and the subscriber would be everyone as well, which I don't think is what the purpose of DDS is for. I might be wrong on this, but there's more.

I also haven't been able to find any detailed explanations on how it all works; all the explanations are very abstract and don't show any proof as to how it accomplishes the properties that it describes. Without knowing the underlying architecture, I am hesitant on using it since I have no idea if the end-product will work as intended. Furthermore, relying on third-party middle-ware can cause licensing issues in the future.