Hey Guys,

So new question. I have implemented a simple ZoneServer in C++ and I am curious as to how to go about handling my ONRPG player replication. So currently, what I do is this:

1) Create and bind a UDP socket to my desired port on the server

2) Wait for incoming data using recvfrom, and when it is received a new thread is created passing that information into it.

3) The thread determines the information within the packet, and if it is a request for a position update, determines the position and starts a new thread "replicate".

4) The replicate thread then replicates the new users position to all clients.

For those who are familiar with Unity, (which is what I am using for my client), I simply have a thread listening for packets and push them to a buffer, and in the main Update loop of my NetworkController I process ~4000 packets from the top of the queue at a time, and Lerp my remote players to their desired position.

The issue is, although there is no lag for the player, there becomes a lot of lag for the remote players on each client when over 3-4 people are connected on the server. Is there any way I can improve my server end?

First, do not create new threads in response to receiving messages.
Second, do not create new threads in response to more/fewer users connecting.
Third, because you only have one network interface, it makes sense to only have one thread that talks to the network.
Fourth, users shouldn't have to "request updates" -- if you're a connected user, you want updates, right?
Fifth, when you say there is "lag," you will need to quantify that.
How much lag?
How many packets are you sending per second?
How many messages are there in each packet?

In general, a network "replication" server might look something like:

int udpsock = socket( ... );
bind(udpsock, ... );
fcntl(udpsock, ... ); // make non-blocking
while (true) {
  int n;
  char buf[1536];
  sockaddr_in sin;
  socklen_t slen = sizeof(sin);
  while ((n = recvfrom(udpsock, buf, sizeof(buf), 0, (sockaddr *)&sin, &slen)) > 0) {
    if (is_new_address(sin)) {
      add_new_remote_player(sin);
    }
    handle_update_from_player(sin, buf, n);
  }
  if (time_now() > last_send_time + tick_interval) {
    last_send_time = time_now();
    n = get_position_of_all_players(buf, 1536);
    send_packet_to_all_players(buf, n);
  }
}

This loop could obviously be significantly improved by using select() with a timeout to not busy-wait, by detecting clients that have stopped sending updates to time them out, etc. But it should get you on the right direction towards building a basic, functional, high-performance repeating server.

Wow hplus, this is like invaluable, so I am in the process of ridding myself of my threads, so in this case my question becomes then that last function, send_packet_to_all_players, is that ONE packet that has information about all players in the zone?

Wow hplus, this is like invaluable, so I am in the process of ridding myself of my threads, so in this case my question becomes then that last function, send_packet_to_all_players, is that ONE packet that has information about all players in the zone?

Send as few packets as possible. You can put multiple messages into a single packet. Each object's position update will often be a single message. Maybe it'll be combined with orientation and velocity data or maybe a single message will include information about a group of objects. It all depends on your game (there is not a single correct answer).

You do not generally need or want to send information about an entire zone to each player. If player A is standing 100 units away from player B and the in-game visibility is only 10 units, why would the players need to know anything about each other? This is generally referred to as "area of interest filtering." Figure out which players care about which objects and only send updates about those objects. This filtering can range from very simple radius checks to some very complex queries, depending on your needs.

Right, thanks Sean! and hplus too!! I have implemented this new server, will conduct my tests tomorrow, hopefully my lag problem will be fixed, and hplus I will try and quantify the data if this does not work for you o.0. Just out of curiosity, why is multithreading looked down upon?

Right, thanks Sean! and hplus too!! I have implemented this new server, will conduct my tests tomorrow, hopefully my lag problem will be fixed, and hplus I will try and quantify the data if this does not work for you o.0. Just out of curiosity, why is multithreading looked down upon?

Multithreading is not the problem here, the usage you described is the problem. Spinning up new threads is something to be avoided because it is exceptionally slow and costly. In your case, it is actually reasonable to use threads but you need to do it only as you determine it is needed and also you don't want to actually "start" threads, they should all exist and never shutdown, they simply idle until there is work.

If/when you get to the point where it makes sense, there are many ways to go about things but I tend to move straight to the OS specifics. There is very little point running your system multithreaded through the generic API's when WinIOCP, ePoll or KEvent API's do a significant portion of the thread communications for you and cut out a fair portion of the overhead involved. Of course, while epoll and kevents are fairly simple and give you great benefits, WinIOCP is a PITA to get correct. Either way though, when you need the threaded solution, the OS specifics cut out a lot of intermediate bits that allow you to reduce latency issues and maintain high performance. But, again, doing this is really only going to be valid for a pretty high amount of traffic/process, it is up to you to decide when to switch over.

threads

In addition to what hplus0603 already said, you should generally never spawn threads, except when your program starts up. And then, you should spawn a fixed amount of them (typically equal to the number of CPU cores, or one less). Then assign tasks to the threads via one or several queues (lockfree ideally, but queues with a lock work just fine too, if you manage tasks properly). Note that when I say "task" then that does not mean something like "add two numbers", but something like "process 1,000 items".

The reason for that is that spawning threads is a lengthy, expensive operation which you do not want to do while receiving (or rather, not receiving, but dropping) UDP packets, and spawning a thread per task is generally a bad design, which is neither efficient nor scales well. Many threads means many context switches, which means a lot of CPU wasted for nothing.

You definitively want receiving UDP traffic to happen on one thread that does nothing else, since if you don't receive datagrams "immediately", your receive buffer will quickly fill up, and you will drop packets. So, you will definitively not want to do anything like reading a file, processing complicated game logic, or spawning threads in that same thread which handles the network. You don't need more than just one thread for that either, though. One thread is absolutely capable of doing that task (even with plain old blocking API calls and nothing special!).

ONE packet that has information about all players in the zone?

This depends. While that may be a very efficient solution (for some rare cases), it may not be the correct one. Every player in the zone (probably) does not see everything, but you are still sending every player the complete information. That may be acceptable, or it may be exploitable and therefore forbidding (depends on your game).

Also, not all information is equally important to each player. Depending on the amount of updates, you may have to send a considerable amount of data to every player. Bandwidth is not only limited (both at your end and at the other end!) but also costs money. You will therefore wish to reduce bandwidth by sending each player only

a) what they can actually see

b) what, in this subset, matters most

c) no more than a fixed so-and-so-much budget per second

It matters big time if someone who is 2 meters away makes a side step or changes clothes. This is immediately obvious. However, changing clothes may not be as important as pulling a gun.

It doesn't matter at all if someone 250 meters away makes a step or changes clothes. You likely won't notice at all.

Since the number of updates that you need to transmit scales at least quadratically with distance (according to the area of a disk for 2D/pseudo-3D, or if it's real 3D the volume of a sphere), you usually need to apply some "importance metric" that is somehow related to distance for each receiving user.

WinIOCP, ePoll or KEvent

This is an excellent tip for TCP, but less for UDP. With TCP, you have many sockets in an "unknown" state, but you can only do one thing at a time, so you need some special magic that tells you which one is ready so you can read from (or which overlapped operation has just finished).

Using UDP, you have a single socket, no more. And that single socket either has something to read, or it doesn't. Instead of blocking on a function that tells you "it's now ready" and then calling recvfrom, you can just as well block on recvfrom, which will conveniently return when something came in. Then push that packet on a queue (making the processing someone else's problem!) and immediately call recvfrom again.

For a FPS game with smallish levels and smallish number of players (such as Quake) sending a single packet with information about all players for each network tick is totally fine, simple, and will perform well. You only need to generate the contents of this packet once per tick, too, which is a bonus.

When the number of players goes up (say, above 30) and the sizes of levels goes up (so not everybody can possibly snipe everybody) then you can start doing interest management, where "close" or "important" entities are sent every network tick, but "far" or "unimportant" entities are sent less often. These packets need to be generated differently for each player, because each player has a different viewpoint.

When it comes to threading, threads are great to use multiple CPU cores. Thus, the ideal program/system has exactly one thread per CPU core. To make sure that those threads always have work to do, you should be using some kind of notified, asynchronous, or non-blocking I/O, so that threads don't get stalled blocking on I/O. For things that don't have convenient asynchronous APIs, like file I/O on Linux, you can spin up an additional thread, which receives requests for I/O, performs the requests, and then responds back, basically implementing async I/O at user level. You'd use some kind of queue between the other threads posting requests, and responses getting queued.

Similarly, there are physical hardware limitations. Each hard disk can only read from one track on the spinning platter (or one sector of flash) at a time. Each network interface can only send one network packet at a time. Thus, having more threads waiting for each particular piece of hardware at the same time is inefficient. Over-threading a program is very likely to run into this problem, where the threads don't give you any performance, but end up costing in resources and complexity (and bugs!)

Now, this is how high-performance servers end up being structured. If you're currently testing with 4 players, chances are that you don't need to implement this structure. You could get away with a single thread for a long while! And, once you start adding threads, adding one thread per subsystem (collision detection, networking, disk I/O, interest management, ...) is generally easier to debug and optimize than trying to add one thread per user, where each thread can potentially "do all the things" and the number of threads is not bounded or even managed.

Wow, I did not even know how this works, learning something new everyday. Sounds like a fun challenge!!! Thanks for all the help, can't wait to see the performance difference once classes are over today. I suppose after this, my team and I will be having a discussion about some of the game mechanics we would like to see in the game, and then how we will go about using all of your advices to restructure the way we are thinking about our server at the moment. But I understand now that having a manageable and bounded # of threads dedicated to certain tasks, which do not wait on the same hardware and utilize ASIO to the best of their capabilities, as well as a queue based system for passing tasks to threads is the best way to go about it. I will post back here once I see how well it worked out. Thanks soo much!!

Just out of curiosity actually, how would one go about not locking the queues as they are being read from and added to in different threads. Isn't that dangerous, at least from my clearly rudimentary understanding of threads, mutex's are required for synchronous communication but I should be going for asynch, however accessing the same memory in 2 places at the same time is dangerous no?

For a FPS game with smallish levels and smallish number of players (such as Quake) sending a single packet with information about all players for each network tick is totally fine, simple, and will perform well.

Well yes, from a pure performance point of view, it's OK for a Quake-style of game (not so for something much bigger, though).

But my point about knowledge remains. In a game where several people compete, it can be troublesome to provide information to people that they actually can't know. Such as those shoot-through-wall aimbots, or other things. Imagine someone implements a minimap where enemies show as little dots (and nobody using the genuine client has such a mini-map). Imagine knowing what weapon, how much armour, and how much health your opponents have (when you shouldn't!), and where they hide. Imagine knowing which door they'll come through before they know (because you can "see through" the door).

No player should ever know the whole world, normally. Not unless it doesn't matter anyway.