I'm finding that broadcasting certain messages to players, like position updates and so forth, is causing me to sit in my message transmitting loop for quite a long time [since i queue up messages and send them all at once], and was wondering if there was an easier way to do this that i'm just overlooking [since I've got good experience in other fields, and can narrow down who things go to correctly, but am just now bringing my projects into the networked world, and am still just getting the hang of things]. I've got my first game with a large player count already under my belt [up to several thousand at a time], but this exact problem always sort of bugged me, even though it didn't damage the game any. Few things I've thought of, have a low end [got about 5 300 mghz sitting around :P, and i can throw network cards in them no problem] machine attached to the main game server that just act as broadcasting stations. Send them the message in the form of a list of identifiers, followed by the message packet to be sent, and have this lesser machine assemble the packets and send them out, with the duplicated data, thus distributing all the waiting around onto machines that are too slow to do the high impact stuff anyways. The two machines sitting right next to eachother would likely make for very little latency. Have this second machine worry about retransmission and all that messy stuff. Also a possible idea, randomly select a subset of the players [like, 40% or so] from each update in the cases when the updated players are above a certain threshhold [80 or more players getting the exact same update], and do the old peer to peer thing with them. This puts things into the hands of the client [which is in general a very very bad thing] but it puts the same task in the hands of what is always a very large number of random players. Have each player who gets the packet, get a list of others who should get it, and have them broadcast it and worry about all that junk. With a large enough sample of players, it should be easy to do a majority rules set on them, and if the one kind of packet is over a certain threshold [like 5%], that differs from the others, the client checks with the game server about what the actual message should be [and reports where the 'bad' data came from, once the correct data is confirmed], that way it only takes one or two legal player around to keep everyone in sinc, and makes it obvious who's machine is putting out bad data. Since the client will have a lot more computing power to spare, compared to the server, the cost won't be that great to do all the comparisons. Also, it'll just be checking server-sent transmissions to keep everyone in sync, so it won't open up for cheating or anything. Anyone else have any ideas? anyone else bothered about the level of repetition in your network traffic?....as someone who optimizes everything within an inch of uselessness, these extra bits flowing out of my main game server literally keep me up at night.

I think your first suggestion (broadcast relay servers) is a better idea. You can't guarantee that your players have a firewall allowing transmission on a separate port / stream (I assume this is what you're talking about), or reception without NAT punch-through.

Your first suggestion, as you touch on, is safer for authoritative data. Personally I think increasing your cluster size is more sensible, especially considering that you're currently broadcasting through the single pipe to your cluster without ill effect, indicating pure bandwidth isn't an issue.

Just my opinion.

We offload view management to a process separate from simulation (call it a layer 5 router if you will). If you do sophisticated view management, that can actually be as much CPU time as the simulation itself.

Now, you have to make sure that the "long time" you're seeing is actually code being run in your view management, or in the network stack of the kernel, before this is an option. If the "long time" spent is actually just blocking on the sending sockets, then doing proper asynchronous or non-blocking sending with user-code queue management is the first thing you need to implement. Else someone connecting with a 300 bps modem will be able to bring your entire server to its knees :-(.