Hi,

I'm programming a multiplayer VR racing game on mobile, and I have implemented a basic multiplayer server.

I've used a naiive implementation that I've previously written for a mobile strategy game.

Algorithmically everything works fine. However I am getting alot of traffic on the network because my racing game has much more frequent updates than a strategy game.

As I said, the implementation is naiive (a brute force attempt to add multiplayer in 24 hours). This means:

• I am using TCP_NODELAY instead of UDP (can I use UDP on mobile networks? I allways assumed it was blocked)
• I send all the other 10 players' positions to each player every 200ms. The racing is in "ghost mode". There is no interaction with the other players, you just see them on riding next to you. So 200ms update rate is not a game issue. But any slower, and the lag becomes really noticable.
• I am using a text based delivery method (easy to debug across a C# unity client and a Java server [my old Centos server cannot run Mono] ). I am assuming this is not too bad because tcp headers are 20 bytes each anyway, and I am transporting 4 strings: 3 floats: x,y,z, and a 64 bit uuid.

Using all of this, i get around 1MB of data use per minute. Since this should run over a mobile network, that's way too high. You should assume I know very little about real-time multiplayer, with that in mind: Are there any obvious tips I should be using to reduce my traffic drastically?

My current guess is to try and reduce the text encoded messages. By replacing the 3 floats with fixed point ints (percision is not important for a ghost), and reducing the uuid to cyclical int. (If I ever have over 4 billion players, I'll be extatic to fix this bug :-) )

There is something I don't understand.

I improved it a little bit by making my text packets smaller.

It is now ~50 bytes payload per player state.

So with a 200ms update I should be sending 50bytes_payload * 5messages_per_second * 60seconds = 15k bytes of payload data per minute + 6k tcp headers (300 messages).

However when I turn ff my WIFI, I see android reporting around 200k data usage per minute in the app info page. That's 10 times as much as 21k.

Is there some overhead I am missing. Could it be that I misunderstood just how bad tcp_no_delay can be?

(yes I know, the state could be smaller if it was binary encoded, but I still want to understand what I m seeing here...)

didnt you say you send all 10 players position to each player every 200ms? Wouldnt that explain the 10 times amount of data you were expecting?

can I use UDP on mobile networks?

Yes, you can (usually -- there's always some odd duck out there.)

A number of important protocols require UDP, ranging from voice-over-IP, to video streaming, to DNS look-ups, to VPN connections.

didnt you say you send all 10 players position to each player every 200ms? Wouldnt that explain the 10 times amount of data you were expecting?

Nope, that traffic was measured with only 2 test players connected.

Yes, you can (usually -- there's always some odd duck out there.) A number of important protocols require UDP, ranging from voice-over-IP, to video streaming, to DNS look-ups, to VPN connections.

But aren't those whitelisted? Or is any old protocol allowed?

is any old protocol allowed?

In general, yes. If you pay for the data, it's in the carrier's best interest for you to be able to use the data.
If you're on some "all you can eat" kind of connection, filtering might make more sense, to reduce "abuse."
Some companies also attempt to filter the common Bittorrent ports and such.

On the topic of traffic filtering: You can't really "whitelist" a protocol as an ISP/operator, because while a specific port number may be "common," people can put whatever data they want into those packets.
That being said, the reverse is also true. Some companies/enterprises used to filter traffic based on ports, so only (say) 53 UDP and 80/443 TCP would be allowed (DNS, and the web,) and games or instant messengers would then run their game traffic on port 53 or 80, to "look like" DNS or HTTP to get through those firewalls.
There also exist "deep packet inspection" firewalls that will stop a connection if it doesn't seem like it's legitimate for the port. Even SSL can be inspected to look for the SSL framing packets, especially in the beginning/setup.

Test locally, use Wireshark to verify the data being sent. If there's some extra traffic you don't want, you'll find it quickly enough.

Regarding reducing the data sent to synchronise positions, there are a bunch of possible approaches that might work, depending on your game:

• Batch up your updates so that you only pay the TCP/IP header cost once for several messages.
• Racing is usually 2D, to some extent - can you simply send p, q coordinates, where p is normalised horizontal along the track (e.g. 0 is on the left barrier, 1 is on the right) and q is how far around the track you are (0 is the start line, 1 is the finish)?
• Switch your IDs to single bytes. You only have 10 players so each player is uniquely identified by a number from 0 to 9. Those UUIDs can be sent once, at the start, if the other players truly need them.
• If you send velocity instead of position, can you send it less often? Perhaps sending 2 velocity updates per second is enough, plus a position update every 2 seconds to prevent it drifting too far (a bit like MPEG encoding and the concept of I-Frames and P-Frames)
• Consider using one-byte floats instead of 4-bytes or more; you probably don't need more than 255 discrete grades of speed or velocity, especially if you're sending position corrections as well.

...

Thanks,

Great tips.

Can't use wireshark as my server is centos. Guess it's time to roll up my sleeves with grep and tcpdump

Can't use wireshark as my server is centos.

Wireshark runs on Linux. For example: http://serverfault.com/questions/404282/how-do-i-install-wireshark-in-red-hat-linux
Also, you can let tcpdump capture pcap files, that can then be opened in Wireshark!

That being said, Real Men (tm) use "sudo tcpdump -i any -s 65536 -XX" and call it good :-)

Test locally, use Wireshark to verify the data being sent. If there's some extra traffic you don't want, you'll find it quickly enough.
Regarding reducing the data sent to synchronise positions, there are a bunch of possible approaches that might work, depending on your game:

• Batch up your updates so that you only pay the TCP/IP header cost once for several messages.
• Racing is usually 2D, to some extent - can you simply send p, q coordinates, where p is normalised horizontal along the track (e.g. 0 is on the left barrier, 1 is on the right) and q is how far around the track you are (0 is the start line, 1 is the finish)?
• Switch your IDs to single bytes. You only have 10 players so each player is uniquely identified by a number from 0 to 9. Those UUIDs can be sent once, at the start, if the other players truly need them.
• If you send velocity instead of position, can you send it less often? Perhaps sending 2 velocity updates per second is enough, plus a position update every 2 seconds to prevent it drifting too far (a bit like MPEG encoding and the concept of I-Frames and P-Frames)
• Consider using one-byte floats instead of 4-bytes or more; you probably don't need more than 255 discrete grades of speed or velocity, especially if you're sending position corrections as well.

Really good advice. I realize this is a big topic and I have read similar advice regarding reducing traffic on websocket connection.

But I am bogged down by the sheer amount of work.

Regarding the player id, I simply use the socket id as the player id but it is combined using numbers and letters which has too many bytes. But if I change it to some simpler form the id on the client side will be different from the ones on the server side as I check whether the player is connected using the socket id.

As with the floating point number, I am using 64 bits floating point which is the standard JavaScript number. It is a double-precision float. When converted to binary data its size is 8 bytes. Hopefully I can find a way to convert it to a single-precision floating point number in JavaScript.

Also JavaScript float number uses IEEE 754 standard, I hope modifying the precision won't affect the real position on browser too much.

Now each of my player sends out about 10KB/s which is far from optimized. I hope that I can reduce it to 1-2KB /s so that I can have a few hundreds happy players on the server.