Hi again.

Working on ways to sync projectile spawning for weapons. I have things more or less figured out for how the client will tell the server that it fired (incrementing a sequence number), but I've got a few options for how to relay the projectile being spawned to all of the other nearby players.

First, some constants:

- Think multiplayer bullet hell arena shooter, though with not quite so many bullets.

- 50Hz tick, 25Hz packet send rate (so 40ms between packets)

- To spawn a projectile, the client needs to know what server tick it happened on, what entity shot it, and what the position/orientation/velocity of the entity was at the time (technically we could infer this but I want to be as precise as possible).

Right now I have two schemes for sending events:

1) Reliable Ordered. Recipient keeps a sequence ID of its last processed event. For every event received, if the event's ID is one more than the sequence Id (not just greater than, but the exact next event), increment the sequence Id and process the event. Send the sequence ID to the sender so the sender can delete old pending outgoing events and resend unreceived ones.

2) Unreliable with optional redundancy. Sender keeps outgoing events in a list. Each event is resent N times (set on a per-event basis). No acking or feedback from the recipient. Recipient keeps an expiring hash set of received events to avoid re-processing duplicate events. Sender has no way of knowing the recipient has received a particular event, so we always resend N times even if the event was received on the first send.

I'm torn between using (1) or (2) (with a long expiration time) for spawning projectiles. In both cases I'm worried about clogging up the pipe if there are a lot of players in one area shooting at each other, and wondering if there's a better way to do it.

One thing I've thought about is a variant of (2) where the recipient sends back the last N (10?) received event IDs so the sender can clean out some of its unreliable events if they have a long expiration timer. I would love to just have the recipient send back the highest event ID it's received for (2) and use that to clean, but that means if the recipient receives packets out of order it may skip some events. The reason I'm hesitant about using (1) is that if the message queue gets really backed up, we could be trying to spawn projectiles that have long since expired.

Just curious if anyone has any suggestions or could share experience about how they handled it themselves.

Whatever solution you use will depend on a few things, including whether your client simulation runs in the past or not. If you're already interpolating entities, that will give you some additional time to receive information about projectiles.

When it comes to entity replication, I will send a reliable un-ordered packet that contains the information required to instantiate the entity client-side. From there, I would send state updates as unreliable, discarding out-of-date packets. You could even reduce bandwidth for predictable projectiles and just send the initial state, with the network tick and velocity, and allow the client to simulate it.

I'm generally uncomfortable with the notion of reliable ordered. The reason most use UDP for fast-paced games is to avoid the latency that is fundamental to the operation of a reliable stream over unreliable network conditions. At least with reliable unordered you can proceed other state whilst the packet is redelivered when dropped.

So I already do pretty much all of that. A little more about my system:

- State updates received from the server are dejittered with a forced delay of around 100ms.

- Entities are interpolated (if we have data) and extrapolated to an extent if we don't.

- The game is fully server-auth with clientside prediction (already implemented).

All of the entity instantiation and state synchronization is already done. This is purely about what to do when the server needs to tell all the other clients (other than the firing one) that an entity has created a projectile. As you say, the projectiles are predictable -- they don't bounce or change course (yet), so all I need to do is synchronize where and when they were created. Because of this, I'm not treating them as entities. I want to just send one message that they were created and then pretty much forget about them until they hit something.

I agree about reliable ordered. I have it as a separate channel and really want to use it as sparingly as possible. However, unreliable alone isn't dependable enough for something like spawning a projectile, which is fairly important.

I would love to just have the recipient send back the highest event ID it's received for (2) and use that to clean, but that means if the recipient receives packets out of order it may skip some events.

http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/

The trick from that article which works well is to send both the most recently received packet ID and a rolling bitfield of the last N receieved packet IDs (e.g. N=32 or the like). That gives the sender a window of N+1 packets and a few dropped ACKs don't result in too much excessive data resending.

Example: if the client receives packets 1, 2, 4, 5, 9, 10 (so 3, 6, 7, 8 were dropped) then it might send back a packet with ack=10 and ack_bits=10001011 (each bit maps to A-B*2^P, where A is the value of ack, P equals the bit position, and B is the bit value).

I would love to just have the recipient send back the highest event ID it's received for (2) and use that to clean, but that means if the recipient receives packets out of order it may skip some events.

http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/

The trick from that article which works well is to send both the most recently received packet ID and a rolling bitfield of the last N receieved packet IDs (e.g. N=32 or the like). That gives the sender a window of N+1 packets and a few dropped ACKs don't result in too much excessive data resending.

Example: if the client receives packets 1, 2, 4, 5, 9, 10 (so 3, 6, 7, 8 were dropped) then it might send back a packet with ack=10 and ack_bits=10001011 (each bit maps to A-B*2^P, where A is the value of ack, P equals the bit position, and B is the bit value).

Exactly the sort of thing I was looking for. Thanks!

Another option is to always include the last N shots fired (time, pos, vel) in each packet. If one is dropped, no big deal; the next packet that makes it will tell everybody about where it started.

As a related question while I'm still working on this:

Right now my reliable event message scheme is pretty simple. I queue all outgoing reliable events on the server. Every send I pack as many reliable events as I can in the packet, starting from the last one the client acked. The client just sends its ack sequence ID, which it increments every time it processes an event (as long as the next expected event has arrived).

This obviously causes the server to resend a lot of messages while it's waiting for the RTT on the client ack. Is there a smarter way to do this?

I know Gaffer talks about this but in his example he's talking about whole packets, which are compound data structures and can be sent in parts. In my case an event (think like an RPC message) is atomic and small, just a couple of parameter values packed together. You either send it or you don't. I could do some prediction where the server assumes the client has gotten its next messages, but I'm wondering if it's worth the mess.

Is there a smarter way to do this?

You could assume that everything will deliver, and just pack more events in, with a sequence number.
On the client side, if you've seen reliable event X, only apply reliable event X+1 -- if you see anything higher sooner, send a NAK back to the server.
Then, the server re-starts the stream from X+1 when it gets the NAK.
The server still needs a backlog of pending messages, and an ACK to know that the messages can be removed.

You can then split reliable messages across different channels, where it's OK to re-order messages between channels, if you want more flexibility.