Today I gathered my group together in which we had 6 of us trying out the networked game play. It didn't go too well. When there is only 2 of us everything works as expected and it's enjoyable. However, some strange discrepancies started occurring when there were 6 of us; 3 or more really. Individuals started to time-out mainly, some were even crashing for some unknown reasons. Mainly tho, the time-outs is what I'm concerned about. That isn't supposed to happen unless they don't receive a reply from the server after 30 seconds or so, or if they disconnect manually.

One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so.

My network system is currently just TCP, which I think may be the root of my problem. I've been thinking that all of my packets should be guaranteed to be delivered timely since we are only working across LAN, I'm starting to become doubtful. However, another concern I suppose is maybe I am sending update messages to frequently.

Right now every .125 seconds and if the player's transform changed I send a message that looks like this:

// send our transform information to the other players
  transMsg << NetworkManager::GetInstance().GetServerCmdFromMessageType(MESSAGE_TYPE::Game)
           << "!SPECIFICOBJECTMESSAGE " << netCompRef.GenerateMessageID("UPDATEOBJTRANSFORM")
           << gameObj->m_Pos.GetX()   << NLDS << gameObj->m_Pos.GetY()   << NLDS << gameObj->m_Pos.GetZ()   << NLDS
           << playerCam->m_Rot.GetX() << NLDS << playerCam->m_Rot.GetY() << NLDS << playerCam->m_Rot.GetZ() << NLDS
           << gameObj->m_Scale << NLDS << "\r\n";

I'm also doing something not good for when we are shooting. Every bullet that is generate will send to the server the creation of the bullet, which since we have rapid fire now is definitely sending a ton of messages to the server, then back to the other connected clients. Also, another thing I was thinking is maybe my router's firewall is detecting the influx of packets and is thinking it's a DDOS or something, which then terminates the connection on that end. However, I don't think that's the case, but it might be possible.

Thanks for any insight in helping me improve my networking, I'm willing to hear any suggestions.

Today I gathered my group together in which we had 6 of us trying out the networked game play. It didn't go too well. When there is only 2 of us everything works as expected and it's enjoyable. However, some strange discrepancies started occurring when there were 6 of us; 3 or more really. Individuals started to time-out mainly, some were even crashing for some unknown reasons. Mainly tho, the time-outs is what I'm concerned about. That isn't supposed to happen unless they don't receive a reply from the server after 30 seconds or so, or if they disconnect manually.

One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so.

My network system is currently just TCP, which I think may be the root of my problem. I've been thinking that all of my packets should be guaranteed to be delivered timely since we are only working across LAN, I'm starting to become doubtful. However, another concern I suppose is maybe I am sending update messages to frequently.

Right now every .125 seconds and if the player's transform changed I send a message that looks like this:

// send our transform information to the other players
  transMsg << NetworkManager::GetInstance().GetServerCmdFromMessageType(MESSAGE_TYPE::Game)
           << "!SPECIFICOBJECTMESSAGE " << netCompRef.GenerateMessageID("UPDATEOBJTRANSFORM")
           << gameObj->m_Pos.GetX()   << NLDS << gameObj->m_Pos.GetY()   << NLDS << gameObj->m_Pos.GetZ()   << NLDS
           << playerCam->m_Rot.GetX() << NLDS << playerCam->m_Rot.GetY() << NLDS << playerCam->m_Rot.GetZ() << NLDS
           << gameObj->m_Scale << NLDS << "\r\n";

I'm also doing something not good for when we are shooting. Every bullet that is generate will send to the server the creation of the bullet, which since we have rapid fire now is definitely sending a ton of messages to the server, then back to the other connected clients. Also, another thing I was thinking is maybe my router's firewall is detecting the influx of packets and is thinking it's a DDOS or something, which then terminates the connection on that end. However, I don't think that's the case, but it might be possible.

Thanks for any insight in helping me improve my networking, I'm willing to hear any suggestions.

are you sending a long string for each message ? if so you should switch to a binary format, a 32 bit float uses 4 bytes, a 32 bit float encoded as a string tends to use 7+ bytes, things like "!SPECIFICOBJECTMESSAGE " is 23 bytes (could probably be replaced by a 1 or 2 byte number), you also get a nice chunk of CPU overhead from the encoding/decoding (converting numeric values to strings and concatenating them is not cheap)

Your timeout / disconnect problems are probably caused by something else though, (on a LAN you should be able to send tons of data without any problems)

I've been able to reproduce the random disconnect bug we were encountering earlier today. It seems whenever more than 2 people join, 3 people in my test case, one of the other clients will get dropped on the server end, but only after an arbitrary amount of time has passed. I was monitoring the time since last ping on my server end and that's never reaching the time-out period. Something else in my server code is doing something strange somewhere. I really like the idea of reducing the size of my sent packet, I am in the process of working on a way to register the particular network messages as enumerations, also to reduce my string comparisons. I hope that will help also.

Rule #1: If you can't measure it, you can't control it. Your first order of business is going to be to get a precise accounting of three things, in this order of importance:

1. What packets are being sent from the server?
2. What packets are being received at the clients?
3. What's happening to the packets in between?

The first two you track the same way: Put code in your networking functions to log all your activity. You definitely want to log the timestamp (frame number is probably good enough) and number of bytes of every individual receive and send; secondly you want to log the message types (so that you can see what kind of traffic is causing problems); thirdly you want to log object IDs (so that you know what specific objects are causing problems); and finally if your logging system has the capacity, it's certainly nice to see the full message contents in the log. But above all you need to make sure that the logging itself doesn't slow your game down and contaminate your data--you need to be smart about logging to memory and committing that to disk in block writes infrequently and asynchronously.

The third you track by using tcpdump or Wireshark to monitor the actual packet traffic. But that's third on the list for a reason--that's probably not your problem. Focus on the first two for now, and just know that you've got options for the third if those don't turn up anything interesting.

I won't randomly suggest things to try in the absence of data, because Rule #2: Premature optimization is the root of all evil. However, here are some things that your logs might reveal that would be problems... but don't rush off and act on any of these unless/until you have data that shows they're your problems:

• Maybe you're only calling receive once per frame. When all your data generated for a given frame fits within a single MTU window (probably 1400ish bytes), everything would work fine. The moment it spilled over into two+ packets per frame, you'd simply fall further and further behind with every single frame.
• "One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so." As you've described it--commands are being sent and you don't know why--this sounds like a bug, and is certainly a red flag. In principle, you should know exactly how every packet in your log got generated.
• "My network system is currently just TCP, which I think may be the root of my problem." This is exactly the kind of assumption you should not leap to in the absence of data. Your initial thought--that a modern LAN or wifi connection should be able to transmit data fast enough and reliably enough that there's basically no difference between TCP and UDP--is correct.
• "However, another concern I suppose is maybe I am sending update messages to frequently." Actually, when you compare the server logs to the client logs, you might discover that one side thinks it's sending lots of little packets frequently, but the other side actually receives big packets infrequently. If I were going to suggest something to try in the absence of data, I would say to make sure you have Nagling turned off, because it's on by default and you almost certainly do not want it.
• "Right now every .125 seconds" You might actually see your game perform better if you increased the update frequency. Measure, make a change, measure again!
• "<< "\r\n";" You should not change your serialization format in the absence of data. That having been said... yeah, once you have data, I think the odds are very good that data will tell you that your text serialization is way too verbose. Clients won't care; it's the poor server, who consumes 5-6x the bandwidth of any one client, who's getting pounded by this. Or rather who may be getting pounded by this--gather data to prove it!
• "Every bullet that is generate will send to the server the creation of the bullet," Okay, if you wanted to make just one change in the absence of any data, go ahead and fix that, just on principle. :) But I think your data will show that that's just a small part of the problem.
• "maybe my router's firewall is detecting the influx of packets... I don't think that's the case" I don't think that's the case either.

So: Go gather data and come back!

I vote that you focus on #1 and #2 of ProfBeeman suggestion. It's really useful to be able to see what is sent over the network :).

For reference material i keep many links in my gamedev notebook at http://www.gamedevpensieve.com/network . Some of them might give you some ideas.

I see you sending, but I don't see you parsing.
TCP does not guarantee that a single call to send() ends up in a single call to recv().
You can get more, or fewer, calls to recv() on the rececing end than you make to send() on the sending end.
Thus, all TCP reception needs to be into a buffer of some sort, and you parse a full message when you have a full message in the buffer.
There will also likely be additional data in the buffer after parsing the full message; save this data because it's the start of the next message (and may be the complete next message, too.)

So, step 1 in debugging would be to make sure that you treat run-together messages correctly.
Step 2 in debugging would be to read whatever switch() statements you have to make sure you didn't lose a "break" somewhere.
(That would easily cause unexpected commands to be decoded.)

Step 3 is to make it possible to record all incoming data on the server or on a client, and later play it back from scratch, to be able to debug whatever problems.
This is actually pretty easy to implement, and super, super valuable. In the end, you can ship this as a "game replay" feature for the client.
You would then capture a gameplay session, and put the data on your development machine. Put a breakpoint in "player requests exit" and Bob's your uncle.

As hplus0603 and ProfBeeman have suggested the send/recv calls don't always align especially when network traffic picks up as you have suggested it does.

Some middleware libraries like 0MQ (ZeroMQ) wrap this for you and provide messaging patterns like you are looking for.