I have a few questions regarding the principle of operating a network tick rate.
Firstly, I do fix my tick rate on clients. However, I cannot guarantee that if the animation code or other system code takes too long, that it will operate at such a frame rate.
To calculate the network tick, I would simply use "(self.elapsed - self.clock_correction) * self.simulation_speed", assuming the correction is the server time + latency downstream.

However, if the client runs at 42 fps, and the simulation speed is 60 fps (or, the server runs at 42 and the simulation is 60), I will eventually calculate the same frame successive times in a row if I round the result of the aforementioned equation. (This was an assumption, it seems unlikely in practice, but this will still occur when I correct the clock). How should one handle this?
Furthermore, should the simulation speed be the same as the server fixed tick rate, for simplicity?

One last question;

If I send an RPC call (or simply the input state as it would be) to the server every client tick (which I shall guarantee to run at a lower or equal tick rate to the server), then I believe I should simply maintain an internal "current tick" on the server entity, and every time the game tick is the same as (last_run_tick + (latest_packet_tick - last_packet_tick)) on the server I pop the latest packet and apply the inputs. This way, if the client runs at 30 fps, and the server at 60, then it would apply the inputs on the server every 2nd frame.

However, if the client's packet arrives late for whatever reason, what is the best approach? Should I introduce an artificial buffer on the server? Or should I perform rewind (which is undesirable for me as I am using Bullet Physics, and thus I would have to "step physics, rewind other entities" and hence any collisions between rigid body and client entities would be cleared"? If I do not handle this, and use the aforementioned model, I will eventually build an accumulation of states, and the server will drift behind the client.
Regards, Angus

I am assuming a fixed time step here. Let's call it 60 simulation steps per second.

If the client cannot simulation 60 times per second, the client cannot correctly partake in the simulation/game. It will have to detect this and tell the user it's falling behind and can't keep up, and drop the player from the game. This is one of the main reasons games have "minimum system requirements."

Now, simulation is typically not the heaviest part of the game (at least on the client,) so it's typical that simulation on a slow machine might take 20% of the machine resources, and simulation on a fast machine might take 5% of machine resources. It is typical that the rest of available resources are used for rendering. Let's say that, on the slow machine, a single frame will take 10% of machine resources to render. With 20% already going to simulation, 80% are left, so the slow machine will display at 8 fps. Let's say that, on the fast machine, a frame takes 1% of machine resources to render. With 95% of machine resources available, the fast machine will render at 95 fps.

Some games do not allow rendering faster than simulation. These games will instead sleep a little bit when they have both simulated a step and rendered a step and there's still time left before the next simulation step. That means the machine won't be using all resources, and thus running cooler, or longer battery life, or being able to do other things at the same time. Also, it is totally possible to overlap simulation with the actual GPU rendering of a complex scene, so the real-world analysis will be slightly more complex.

Note that, on the slow machine, simulation will be "bursty" -- by the time a frame is rendered, it will be several simulation steps behind, and will simulate all of those steps, one at a time, to catch up, before rendering the next frame. This is approximately equivalent to having a network connection that has one frame more of lag before input arrives from the server.

When a client sends data to the server such that the server doesn't get it in time, the server will typically discard the data, and tell the client that it's sending late. The client will then presumably adjust its clock and/or lag estimation such that it sends messages intended for timestep T somewhat earlier in the future. This way, the system will automatically adjust to network jitter, frame rate delays, and other such problems. The server and client may desire to have a maximum allowed delay (depending on gameplay,) and if the client ends up shuffled out to that amount of delay, the client again does not meet the minimum requirements for the game, and has to be excluded from the game session.

Have you any ideas for a robust tick synchronisation algorithm? I have found mine tends to be unstable

I have described a robust tick synchronization mechanism more than once before in the last two months on this forum.

I have described a robust tick synchronization mechanism more than once before in the last two months on this forum.

I have seen your posts, and they have been helpful. I have also found that I seem to get an unreliable jitter no matter what step I use to modify the client clock. I use an RPC call that "locks" a server variable until the client replies with acknowledgement of the correction, so I avoid bouncing around a median value because of latency.

I use an RPC call that "locks" a server variable until the client replies with acknowledgement of the correction, so I avoid bouncing around a median value because of latency.

I would not expect that to be robust.

First, I would want the server to be master, and never adjust its clock.
Second, because of network jitter, you will need some amount of de-jittering. You can then use your clock synchronization to automatically calibrate the amount of de-jitter needed!

When a client receives a packet that contains data for some time step that has already been taken, bump up the estimate of the difference.
When a client receives a packet that contains data for "too far" into the future, bump down the estimate of the difference.
The definition of "too far" should be big enough to provide some hysteresis -- probably at least one network tick's worth (and more, if you run at high tick rates.)

Finally, the server should let the client know how early/late arriving packets are on the server side. If the server receives a packet that's late, it should tell the client, so the client can bump it's estimate of transmission delay (which may be different from clock sync delta.) Similarly, if the server receives a packet that is "too far" into the future, let the client know to bump the other way.

As long as the hysteresis in this system is sufficient -- the amount you bump by, and the measurement of "too far," -- then this system will not jitter around a median, but instead quickly settle on a good value and stay there, with perhaps an occasional adjustment if the network conditions change significantly (rare but possible during a session,) or there is significant clock rate skew (very rare.)

Since I have gotten a ton of help from a lot of people on this forum over the years, and especially hplus0603, I figured I could give some back and provide the code for the system for tick synchronization and simulation that I have come up with. This is by no means anything revolutionary or different from what everyone else seem to be doing, but maybe it will provide a good example for people learning. For reference I am using this for an FPS game.

First some basic numbers:

My simulation runs at 60 ticks per second on both the server and client. While it could be possible to run the server at some even multiplier of the client (say 30/60 or 20/60 or 30/120, etc.) I have chosen to keep both simulations at the same rate for simplicity's sake.

Data transmission rates are also counted in ticks and happen directly after a simulation tick, the clients send data to the server 30 times per second (or every 2nd tick), the server sends data to the clients 20 times per second (or every 3rd tick).

Here are first some constants we will use throughout the pseudo code:

#define SERVER_SEND_RATE 3
#define CLIENT_SEND_RATE 2
#define STEP_SIZE 1f/60f

I use the canonical fixed simulation/variable rendering game loop, which looks like this (C-style pseudo code):

float time_now = get_time();
float time_old = time_now;
float time_delta = 0;
float time_acc = 0;

int tick_counter = 0;
int local_send_rate = is_server() ? SERVER_SEND_RATE : CLIENT_SEND_RATE;
int remote_send_rate = is_server() ? CLIENT_SEND_RATE : SERVER_SEND_RATE;

while (true) {
    time_now = get_time();
    time_delta = time_now - time_old;
    time_old = time_now;

    if (time_delta > 0.5f)
        time_delta = 0.5f;

    time_acc += time_delta;

    while (time_acc >= STEP_SIZE) {
        tick_counter += 1;

        recv_packets(tick_counter);
        step_simulation_forward(tick_counter);

        if ((tick_counter % local_send_rate) == 0) {
            send_packet(tick_counter);
        }

        time_acc -= STEP_SIZE;
    }

    render_frame(time_delta);
}

This deal with the local simulation and the local ticks on both the client and the server, now how do we deal with the remote ticks? That is how do we handle the servers ticks on the client, and the clients ticks on the server.

The first key to the puzzle is that the first four bytes in every packet which is sent over the network contains the local tick of the sender, and it's then received into a struct which looks like this:

struct Packet {
    int remoteTick;
    bool synchronized;
    char* data;
    int dataLength;
    Packet* next;
}

Exactly what's in the "data" pointer is going to be very game specific, so there is no point in describing it. The key is the "ticks" field, which is the local tick of the sending side at the time the packet was constructed and put on the wire.

Before I show the "receive packet" function I need to show the Connection struct which encapsulates a remote connection, it looks like this:

struct Connection {
    int remoteTick = -1; // this is not valid C, just to show that remoteTick is initialized to -1
    int remoteTickMax = -1;

    Packet* queueHead;
    Packet* queueTail;

    Connection* next;

    // .. a ton of more fields for sockets, ping, etc. etc.
}

The Connection struct contains two fields which are important too us (and like said in the comment, a ton more things in reality which are not important for this explanation): remoteTick, this is the estimated remote tick of the remote end of this connection (the first four bytes we get in each packet). queueHead and queueTail which forms the head/tail pointers of the currently packets in the receive buffer.

So, when we receive a packet on both the client and the server, the following code executes:

void recv_packet(Connection c, Packet p) {
    // if this is our first packet (remoteTick is -1)
    // we should initialize our local synchronized remote tick
    // of the connection to the remote tick minus remotes send rate times 2
    // this allows us to stay ~2 packets behind the remote on avarage, 
    // which provides a nice de-jitter buffer. The code for
    // adjusting our expected remoteTick is done somewhere else (shown
    // further down)

    if (c->remoteTick == -1) {
        c->remoteTick = p.remoteTick - (remote_send_rate * 2);
    }

    // deliver data which should be "instant" (out of bounds
    // with the simulation), such as: reliable RPCs, ack/nack 
    // to the remote for packets, ping-replies, etc.
    deliver_instant_data(c, p);

    // insert packet on the queue
    list_insert(c->queueHead, c->queueTail, p);
}

Now, on each connection we will have a list of packets in queueHead and queueTail, and also a remoteTick which gets initialized to the first packets remoteTick value minus how large of a jitter-buffer we want to keep.

Now, inside the step_simulation_forward(int tick) function which moves our local simulation forward (the objects we control ourselves), but we also integrate the remote data we get from our connections and their packet queues. First lets just look at the step_local_simulation function for reference (it doesn't contain anything interesting, but just want to show the flow of logic):

void step_simulation_forward (int tick) {
    // synchronize/adjust remoteTick of all our remote connetions
    synchronize_connection_remote_ticks();

    // de-queue incomming data and integrate it
    integrade_remote_simulations();

    // move our local stuff forward
    step_local_objects(tick);
}

The first thing we should do is to calculate the new synchroznied remoteTick of each remote connection. Now this ia long function, but the goals are very simple:

To give us some de-jittering and give us smooth playback, we want to stay remote_send_rate * 2 behind the last received packet. If we are closer to the received packet then < remote_send_rate or further away then remote_send_rate * 3, we want to adjust to get closer. Depending on how far/close we are we adjust one or a few frames up/down or if we are very far away we just reset-completely.

void synchronize_connection_remote_ticks() {
    // we go through each connection and adjust the tick
    // so we are as close to the last packet we received as possible

    // the end result of this algorithm is that we are trying to stay
    // as close to remote_send_rate * 2 ticks behind the last received packet

    // there is a sweetspot where our diff compared to the last
    // received packet is: > remote_send_rate and < (remote_send_rate * 3) 
    // where we dont do any adjustments to our remoteTick value

    Connection* c = connectionList.head;

    while (c) {

        // increment our remote tick with one (we do this every simulation step)
        // so we move at the same rate forward as the remote end does.
        c->remoteTick += 1;

        // if we have a received packet, which has not had its tick synchronized
        // we should compare our expected c->remoteTick with the remoteTick of the packet.

        if (c->queueTail && c->queueTail->synchronized == false) {

            // difference between our expected remote tick and the 
            // remoteTick of the last packet that arrived
            int diff = c->queueTail->remoteTick - c->remoteTick;

            // our goal is to stay remote_send_rate * 2 ticks behind
            // the last received packet

            // if we have drifted 3 or more packets behind
            // we should adjust our simulation slightly

            if (diff >= (remote_send_rate * 3)) {

                // step back our local simulation forward, at most two packets worth of ticks 
                c->remoteTick += min(diff - (remote_send_rate * 2), (remote_send_rate * 4));

            // if we have drifted closer to getting ahead of the
            // remote simulation ticks, we should stall one tick
            } else if (diff >= 0 && diff < remote_send_rate) {

                // stall a single tick
                c->remoteTick -= 1;

            // if we are ahead of the remote simulation, 
            // but not more then two packets worth of ticks
            } else if (diff < 0 && abs(diff) <= remote_send_rate * 2) {

                // step back one packets worth of ticks
                c->remoteTick -= remote_send_rate;

            // if we are way out of sync (more then two packets ahead)
            // just re-initialize the connections remoteTick 
            } else if (diff < 0 && abs(diff) > remote_send_rate * 2) {

                // perform same initialization as we did on first packet
                c->remoteTick = c->queueTail->remoteTick - (remote_send_rate * 2);

            }

            // only run this code once per packet
            c->queueTail->synchronized = true;

            // remoteTickMax contains the max tick we have stepped up to
            c->remoteTickMax = max(c->remoteTick, c->remoteTickMax);
        }

        c = c->next;
    }
}

The last piece of the puzzle is the function called integrade_remote_simulations, this function looks as the packets available in the queue for each connection, and if the current remote tick of the connection is >= remote_tick_of_packet - (remote_send_rate - 1).

Why this weird remote_tick comparison? Because if the remote end of the connection sends packets every remote_send_rate tick, then each packet contains the data for remote_send_rate ticks, which means the tick number of the packet itself, and then the ticks at T-1 and T-2.

void integrade_remote_simulations() {
    Connection* c = connectionList.head;

    while (c) {
        while (c->queueHead && c-remoteTick >= c->queueHead->remoteTick - 2) {
            // integrate data into local sim, how this function looks depends on the game itself
            integrade_remote_data(c->queueHead->data);

            // remove packet
            list_remove_first(c->queueHead, c->queueTail);
        }
        c = c->next;
    }
}

I hope this helps someone

Thank you for all of your support. I hate to admit, but I'm still finding it hard to organise my thoughts on this matter.

At present, I have the following structure for the network code

class Network:
    _interfaces = [ConnectionInterface(address)...]

    def receive(self):
        for data in self.in_data:
            interface = self.get_interface(data.address)
            interface.receive(data.payload)

class ConnectionInterface:
    def receive(self, payload):
        if payload.protocol == HANDSHAKE_COMPLETE and not self.has_connection():
            self.create_connection()
            return

        self.update_ack_info()
        self.update_other_stuff()

        if not self.tick_is_valid(payload.tick):
            self.handle_invalid_tick()
        else:
            self.dejitter(payload.tick, payload)

    def tick_is_valid(self, tick):
        return tick >= WorldInfo.tick 

    def update(self):
        data = self.dejitter.find_latest(WorldInfo.tick)              
        if data:
            self.connection.receive(data)

class Connection:
    def receive(self, data):
        self.update_replication_data()
  

My first concern; should I drop packets which are received at the wrong time (ie, too late or too early?) If so, I won't ACK them in order to avoid a reliable packet failing to retransmit.

As well as this, I will need to keep the client informed of how far from the correct time they are. If I send a reply packet, I feel like i should be handling this in the connection class rather than the interface (the interface simply manages the connection state, whilst most data should derive from the connection itself).

You can then use your clock synchronization to automatically calibrate the amount of de-jitter needed!

What should I infer from the clock difference? Do you mean to say that I should take some indication of the jitter time needed from the magnitude of difference from the average RTT that clock synchronisation determines?

Overall, would I be correct in the following understanding?

1. We de-jitter packets by enough time that we always have (at least) one in the buffer
2. We infer the de-jitter time by the clock sync oscillation about a median/mean value.
3. We then look at the packets we read from the dejitter buffer. If they're early or late we selectively drop them, do not ACK them and inform the client that their clock is wrong (in a certain direction).

I see one other issue; build up in the de-jitter buffer. If we shorten the de-jitter time, we will have already ACKed the packets in the buffer. But we will not be processing them, in order to shorten the buffer size. So how does one respond to that?

For the record, my de-jitter buffer:

from collections import deque


class JitterBuffer:

    def __init__(self, delay=None):
        self.buffer = deque()
        self.delay = delay
        self.max_offset = 0

    def store(self, tick, item):
        self.buffer.append((tick, item))

    def retrieve(self, tick):
        delay = self.delay

        tick_, value = self.buffer[0]
        #print(tick_)
        if delay is not None:
            projected_tick = tick_ + delay
            tick_difference = tick - projected_tick

            if tick_difference < 0:
                raise IndexError("No items available")

            elif tick_difference > self.max_offset:
                pass

        self.buffer.popleft()
        return value

We de-jitter packets by enough time that we always have (at least) one in the buffer
We infer the de-jitter time by the clock sync oscillation about a median/mean value.
We then look at the packets we read from the dejitter buffer. If they're early or late we selectively drop them, do not ACK them and inform the client that their clock is wrong (in a certain direction).

That seems reasonable, except the "not ack" part.
I assume that a single network packet contains a single "sent at" clock timestamp.
Further, I assume it may contain zero or more messages that should be "reliably" sent.
Further, I assume it will contain N time-step-stamped input messages from time T to (T+X-1) where X is the number of time stamps per network tick.

Now, reliable messages can't really have a tick associated with them, unless all of your game is based on reliable messaging and it pauses for everyone if anyone loses a message (like Starcraft.)
So, why wouldn't you ack a reliable message that you get?

When it comes to shortening the de-jitter buffer, I would just let the messages that are already in the buffer sit there, and process them in time. It's not like you're going to shorten the buffer from 2 seconds to 0.02 seconds in one go, so the difference is unlikely to matter.

Also, you don't need for there to "always" be one packet in the buffer; you just need for there to "always" be input commands for timestep T when you get to simulate timestep T. In the perfect world, with zero jitter and perfectly predictable latency, the client will have sent that packet so it arrives "just in time." All you need to do to work around the world not being perfect, is to send a signal to the client each time you need input for time step T, but that input hasn't arrived yet. And, to compensate for clock drift, another signal when you receive input intended for time step T+Y where Y is some large number into the future.

Finally, I don't think you should keep the network packets themselves in a de-jitter buffer. I think you should decode them into their constituent messages as soon as you get them, and extract the de-jitter information at that time. The "queuing" of "early" messages will typically happen in the input-to-simulation pipe.

We de-jitter packets by enough time that we always have (at least) one in the buffer
We infer the de-jitter time by the clock sync oscillation about a median/mean value.
We then look at the packets we read from the dejitter buffer. If they're early or late we selectively drop them, do not ACK them and inform the client that their clock is wrong (in a certain direction).

That seems reasonable, except the "not ack" part.
I assume that a single network packet contains a single "sent at" clock timestamp.
Further, I assume it may contain zero or more messages that should be "reliably" sent.
Further, I assume it will contain N time-step-stamped input messages from time T to (T+X-1) where X is the number of time stamps per network tick.

Now, reliable messages can't really have a tick associated with them, unless all of your game is based on reliable messaging and it pauses for everyone if anyone loses a message (like Starcraft.)
So, why wouldn't you ack a reliable message that you get?

When it comes to shortening the de-jitter buffer, I would just let the messages that are already in the buffer sit there, and process them in time. It's not like you're going to shorten the buffer from 2 seconds to 0.02 seconds in one go, so the difference is unlikely to matter.

Also, you don't need for there to "always" be one packet in the buffer; you just need for there to "always" be input commands for timestep T when you get to simulate timestep T. In the perfect world, with zero jitter and perfectly predictable latency, the client will have sent that packet so it arrives "just in time." All you need to do to work around the world not being perfect, is to send a signal to the client each time you need input for time step T, but that input hasn't arrived yet. And, to compensate for clock drift, another signal when you receive input intended for time step T+Y where Y is some large number into the future.

Finally, I don't think you should keep the network packets themselves in a de-jitter buffer. I think you should decode them into their constituent messages as soon as you get them, and extract the de-jitter information at that time. The "queuing" of "early" messages will typically happen in the input-to-simulation pipe.

Firstly, I send inputs as RPC calls every frame, whilst state updates are sent to the client at a network tick rate.

With regard to the question about build up, I think I was thinking incorrectly about my own implementation.

Are you advising that the client's clock correction also factors in the jitter buffer length, rather than simply accounting for the offset server side?

My current code looks as follows:

class PlayerController:

	def on_initialised(self):
		super().on_initialised()

		# Client waiting moves
		self.waiting_moves = {}
		
		# Server received moves
		self.received_moves = {}
		self.maximum_offset = 30
		self.buffer_offset = 0
	
	def apply_move(self, move):		
		inputs, mouse_x, mouse_y = move
		blackboard = self.behaviour.blackboard

		blackboard['inputs'] = inputs
		blackboard['mouse'] = mouse_diff_x, mouse_diff_y

		self.behaviour.update()
	
	def client_missing_move(self) -> Netmodes.client:
		pass
	
	def player_update(self, delta_time):
		mouse_delta = self.mouse_delta
		current_tick = WorldInfo.tick
		inputs, mouse_x, mouse_y = self.inputs, mouse_delta[0], mouse_delta[1]

		self.received_moves[current_tick] = inputs, mouse_x, mouse_y
		self.server_receive_inputs(current_tick, inputs, mouse_x, mouse_y)

	def server_receive_inputs(self, move_tick: TypeFlag(int,
										max_value=WorldInfo._MAXIMUM_TICK),
									inputs: TypeFlag(inputs.InputManager,
										input_fields=MarkAttribute("input_fields")),
									mouse_diff_x: TypeFlag(float),
									mouse_diff_y: TypeFlag(float)) -> Netmodes.server:
		
		tick_offset = WorldInfo.tick - move_tick
		# Increment difference
		if tick_offset < 0:
			self.buffer_offset += 1
		# Decrement difference
		elif tick_offset > self.maximum_offset:
			self.buffer_offset -= 1
		self.received_moves[move_tick] = inputs, mouse_diff_x, mouse_diff_y
	
	def update(self):
		requested_tick = WorldInfo.tick - buffer_offset
		try:
			move = self.received_moves[requested_tick]

		except KeyError:
			self.client_missing_move()
			return
		
		self.apply_move(move)

With client_missing_move, should that add a tick offset to the clock synchronisation client-side? Because the client is already "guessing" the RTT time.

Also, your previous post confuses me as to whether you mean buffer size or clock offset, I assume you mean buffer size.

When a client receives a packet that contains data for some time step that has already been taken, bump up the estimate of the difference.
When a client receives a packet that contains data for "too far" into the future, bump down the estimate of the difference.
The definition of "too far" should be big enough to provide some hysteresis -- probably at least one network tick's worth (and more, if you run at high tick rates.