When learning about online multiplayer programming I always read that it is important to keep the bandwidth usage low. There are tons of articles that discuss bandwidth optimisations and limitations. However, while developing Awesomenauts we learned that packet count can in some cases be equally important. Somehow I rarely read about that anywhere, so I figured it was about time to write a blogpost about this:

The importance of packet count

I am curious: does anyone here share my experience that sending over 100 packets per second is problematic for some players, while below 100 packets per second the packet count does not matter all that much?

most games send from 10-30 update packets per second, because packets will start to be dropped by some connections if you go too high. The real goal is getting the number as LOW as possible without noticeably degrading game play.

Thanks for sharing!

On the sending (and receiving) machines, there are some overheads in large numbers of packets, depending on how good the drivers and hardware are. I seem to recall that you do fully-connected peer-to-peer, so number of packets will likely grown much faster than for a client/server topology.

Different networks work differently. For example, on some networks, it may be that once you "have the aether" you can send a little or a lot with approximately the same cost. I would imagine this to be more marked on WiFi networks, but it would probably be noticeable on most kinds of networks to some extent.

Did you get any commonality among the networks that have the problems you're talking about? Are they WiFi? Are they using file sharing at the same time? Are they some particular ISP or part of the world? A little more data on what you saw might make it possible to get a little more clarity on what's going on.

I seem to recall that you do fully-connected peer-to-peer, so number of packets will likely grown much faster than for a client/server topology.

Yup. Often there is a shortcut 'same as previous" version that is both smaller and faster.

Even so, the OP's 100 packets per second seems quite high. As the article hints, there is much more than what you see. Your little message is wrapped in a UDP packet (at least 8 bytes) which is wrapped in an IP packet (at least 20 bytes) which is wrapped in an Ethernet payload of 18 bytes plus a minimum of 46 bytes for the payload, and once it moves over past your local router into the phone company it can hit an ATM network with 53 bytes of overhead plus the internal point-to-point protocol (PPP) wrappings, or various other protocols at the data-link layer.

When you can have 100+ bytes of overhead for each message, sending tiny packets can completely destroy your signal-to-noise ratio. If you're sending 100 packets per second, or one per millisecond, there isn't much that changes THAT rapidly. If your home router is using PPPoA transport for those 100 messages you've just transmitted about 10KB of data just on headers and wrappers, so whatever payload you are sending across those 100 messages better be worth the cost.

As I said, I think Awesomenauts does full-mesh peer-to-peer, which means that, in an 8 player game, for each network tick, you have to send each packet 7 times (and receive another 7 packets.) That would explain the large packet count.

(Sorry I didn't reply earlier: I thought I had followed this topic to receive updates but apparently I hadn't.)

Awesomenauts indeed requires that many packets per second because it is fully connected peer to peer. There are six players in a match so everything needs to be sent to 5 other players. It is also a fast game, requiring updating more often. I am quite happy that we managed to get the packet count down to 100 per second in the first place. But yeah, it is still a lot...

Did you get any commonality among the networks that have the problems you're talking about? Are they WiFi? Are they using file sharing at the same time? Are they some particular ISP or part of the world? A little more data on what you saw might make it possible to get a little more clarity on what's going on.

Nope, unfortunately we don't have such data. Friends of ours at an AAA studio are even tracking router types for every user, but our metrics are a lot simpler. It does seem like WIFI is the most common cause for such problems, but having crappy internet somewhere in the jungle also seems to do the trick... ;)