I'm creating a networked game with a large object count and was planning to integrate in the bullet physics engine. However, I have been using server corrected client prediction to give clients smooth movement, rewinding and replaying to synchronise.

The only way I can think of doing this in a physics engine like Bullet, which as far as I can tell can only step the whole simulation at a time, is to either store the positions of every object in the whole sim for some period of time and snap hundreds/thousands of objects back, or to handle motion of objects manually, which partly defeats the purpose of using the physics engine in the first place.

How could I handle this kind of thing?

You don't need to snap every object at once. You can snap only some objects, closer to the player with higher frequency, for example.

Similarly, you can know which objects are "touched" by the player entity, and thus are likely to be out of sync, and prioritize snapping those.

Finally, when you "snap," you can remember the position/orientation of objects on the client, too, so you have something like:

client position at time 120: 32, 18
server position at time 120: 28, 19
client position at time 130: 40, 5

Now, the client receives "at time 120, the position was 28,19."

What the client does is calculate a delta (28-32 = -4, 19-18 = +1) and applies that to the current state (40-4=36, 5+1=6)

This works best for position/orientation, you may want to do something more server-authoritative for velocity and spin. At least if the difference is big, it's probably better to apply the server value wholesale to velocity and spin, although if the delta is small, just applying the delta as per position would be OK.

Finally, yes, this is one of many reasons why most large shooter games don't have tons of important interactive physics in their big maps ?

Just to add to what hplus said, it very much depends on the nature of your game.

Generally the easiest solution is to simulate the physics for anything that isn't a player on the server, and the server only, so by definition what the clients see is an accurate, but delayed version of what is on the server, and no need for rewind.

Of course this breaks down when the player interacts with these physics objects, in the same way that player-player interactions often produce glitches in multiplayer, because of the time delay.

There is also the issue of how much data needs to be sent in the packets for large numbers of objects. Glenn Fiedler has some examples of cutting down the data for this kind of thing:

https://gafferongames.com/

Totally guessing at the nature of your game, as you have not specified, but perhaps you can only try and do client side prediction for the objects that are very close to the player? But of course there will be inaccuracies when these client predicted objects interact with non predicted objects etc.

You really need to give more information on the nature of the game. What are these 100/1000 objects? Are they close to the player? Are they boxes? Fish? Particles?

Are you running the clients ahead of the server (clients rewind and replay to extrapolate game state) or server ahead of the clients (server can rewind and replay to compensate for latency, clients interpolate game state)?

Sorry yeah I should have called the thread something like server->client Reconciliation rather than prediction!

To clarify the game a little, it’s a kind of arcade shooter at the moment. The objects are primarily flocks of enemies, so they aren't full on tumbling physics objects but they are bumping up and moving around each other in hordes. I was also hoping to have them interacting smoothly with some moving geometry too (plus I need a good optimised spatial partition for querying).

“Are you running the clients ahead of the server (clients rewind and replay to extrapolate game state) or server ahead of the clients (server can rewind and replay to compensate for latency, clients interpolate game state)?”

The client is behind the server by lag + small jitter buffering time. The client is executing their own ‘inflight’ inputs locally before server confirmation for responsive movement. The client is, in effect, in a slightly different timezone to the rest of the simulation. The client is running the simulation locally to fill in between sync messages as described in the gaffer on games articles

The client movement scheme I have been using so far is the kind described here:

https://www.gabrielgambetta.com/client-side-prediction-server-reconciliation.html

So my primary issue is the rewind->replay of a client's inputs when the server corrects their position specifically. I can snap back for the correction but the client is then moved forward again for all user input sent since the timestamp of the correction and this is the area I don’t understand.

Hello again, I'm back looking at this problem. Anyone have any suggestions after my clarifications? I' m still kinda stumped...

this kind of sounds like you mix some stuff weirdly - if the client is behind the server in time, why do you need to snap rewind->replay the client? if the client is behind the server, you interpolate/snap from known states - if the client is ahead of the server, you rewind/replay from known (past) states. 

it sound like you mix client-rewind with client-behind-server. 

also - arcade physics with lots of enemies that bump into each other is going to be messy with high latency. note how arcade shooters usually don't do enemy/enemy collisions, not even player/enemy collision-responses, just detection. you might save yourself from a world of pain by rethinking your design.

collisions, especially complex ones with multiple forces, restitution, friction ... can diverge quickly even through minimal differences in starting state. does "bouncing off stuff" really add to your game? you can detect collision just fine without bothering to do realistic bouncing off/collision-responses. 

The client is applying their own inputs locally in order to get instance feedback, before they are processed by the server. The rewind process is when the server tells the player that the position they said they were at in frame X was incorrect. The player now snaps back to frame X and replays all their inputs transmitted since, to allow their local inputs to continue smoothly. But applying inputs means stepping the player's avatar in the physics engine. From what I've read so far it is common technique to give players instant feedback since games like Quakeworld. But I can't see how this is done in physics engines which only have single monolithic steps?

I agree about the direct collision being a bad idea btw. I will shift to handling it via AI behaviour

Caveat it is many years since I did any multiplayer stuff so my memory is hazy and I defer to the others on this / might be wildly wrong:

One key thing to make crystal clear is whether you are using the simpler scheme of displaying every game object that is not the player as a delayed version, I am assuming this. There are some more complex schemes where you are trying to display client predicted versions of other players (I believe Hodgman uses this in his racing game), and this is a whole other ballgame.

In general limiting the amount of objects to be snapped back and resimulated sounds like something you need to design the game around.

Simplest version you could have a local bullet simulation that only included the player, and the static (ground, walls etc) and non-simulated elements (say deterministic moving platforms) of the world. That way your client prediction would tend to match the server except in cases of interaction. Putting more objects into the client simulation is just a way of trying to make it less likely that the server and client simulations will diverge (or rather diverge over a large amount).

In a typical multiplayer game you might also decide to simulate 8-16 other players while doing a client side correction (in some way, perhaps more limited).

Even with 8-16 players, (and especially with more dynamic objects) you might try and simulate only the nearest / most relevant ones .. maybe only 3 could influence the player on a particular tick? For instance if you have a potentially visible set you might only be interested in simulating objects that are within the PVS for the player. The general idea is that stuff that is further from the player or deemed less relevant is less likely to cause a change to the player physics.

Presuming you are using the simpler scheme of displaying on the client a smoothed / delayed version of the server simulation (for everything but the player) then understand that you might end up with 2 distinct versions of non main player objects :

• a client predicted physics version for use only in determining your players position
• a displayed version which is delayed and matches the server simulation, and is not the client predicted version

As to how you go from a current (wrong) predicted player position to the recalculated position to prevent a snap, you just use interpolation of some sort to smooth it out. Although I don't think this is what you were asking.

15 hours ago, zqf said:

But applying inputs means stepping the player's avatar in the physics engine. From what I've read so far it is common technique to give players instant feedback since games like Quakeworld. But I can't see how this is done in physics engines which only have single monolithic steps?

Well, unless the physics engine has specific support for this, a step is just a step. You would remove objects from the physics that weren't relevant, add ones that are, set their positions, velocities etc, then step them, and read the results. As to how you do this in bullet I have no experience, but I'm sure it would be possible to step manually or abuse to do this.

Sorry Lawnjelly somehow I missed your response, thanks for replying.

I think I understand now that it is something that is designed or coded around somehow, boiled down to two options.

> By reducing the number of objects you are simulating so that an entire rewind isn't a big deal.

> By treating players as separate objects that are manually manipulated (either by handling the collision responses of these objects yourself or hacking the physics engine to step individual objects).

I think I will try and handle the collision responses myself as it fits what I have in mind for the game better.

Cheers