Sorry, I accidentally logged into some old account of mine from the mobile, that I didn't know I had here :O So michael20 is me.

Resimulation needs to happen in this case. In MK, they just have a Frame Budget of max 7 rollback frames and when You exceed this limit, The game disconnects the players. 7 Frames supported about 80% of matches played though, so it was good enough.

I Hope Your progress with the game is going well! I am a fan of rollback and I would love to have more people harness deterministic netcodes in Unreal Engine and document developing them!

You probably have seen this video, but here they explained everything, potential optimizations even:

As far as I remember, this video talks about the peer-to-peer solution, which usually cuts away a lot of latency. That's not the case in my situation as I went with client-server architecture.
I am finding it hard to believe, that 7 frames rollback is possible in client-server situation. I may be wrong, but having 60hz fixed time step, with client running ahead of the server by about 1 frame, it would be extremely difficult to have data from server being received from 7 frames to the past.
If fixed frame time is 16ms(60fps), then 7 frames is worth 112ms. Well yeah, if you have ping below 112ms, you're completely covered. However, I want to allow higher lag for my game. I don't see situation where player with network spiking to 200 is disconnected from the game. Even above 200 is completely possible.

So far my progress went good. Since my game is fairly similar to Crossout, for now I went with fixed time step of 30hz. Crossout uses 30hz and their networking is really, really good. This precision introduces no problems regarding my project, so I am perfectly happy with it. And my rollbacks are not that awfully expensive as 30hz is not that much, so in my case 7 frames covers 210 ping already, which is minority of the players. Average ping goes from 60 to maybe 100, which usually account to up to 3 rollback frames, which is something you can't really feel.

In 2001, we did 50 frames of rollback at 30 Hz simulation rate for There.com. We ran a 20 person virtual world scene on a 56k modem, using mostly-deterministic simulation. One helpful trick at the time was to keep a log of all the states for the objects for the steps, and which other objects they interacted with, so we didn't need to re-simulate an object that hadn't interacted with something that got corrected.

We also did time dilation; objects seen further away from the player were put further back in time (higher ping) which made them less likely to be corrected. I think we even got a patent on parts of that at the time, but I'd expect that to have expired by now. You might be able to look up the patent filing to read up on how it was done – that's the whole point of a patent: teach the world how to do it, so that after patent expiry, the knowledge is not lost!

Nice info!
For now it will be impossible for me to do any kind of “local” resimulation. I am running everything on a single PhysX scene, which only allows me to go back and forth in time on the entire scene, I have no way of resimulating individual physical objects. However, that might be an option in the future, I could come up with some system to break this stuff into multiple scenes, maybe by dividing world into sectors or some other cool stuff. For now this is out of scope of what we are doing.

That thing to put further away players a bit more back in time would work great. However, in my game I can fire at far away vehicles too, which could be an issue to make it fair to everyone in the game. But I may try it out when optimization part kicks in.

Thank you for the ideas! ?

I'm coming into this late, but I did want to add that the traditional “rollback and replay” system was designed to be applied to a single locally-controlled entity where mispredictions are rare and where the physics demands are relatively simple -such as an FPS where it's mostly just moving a player capsule around. It's easy to do many frames of replaying if you're only really simulating one object!

If you have a complex local physics simulation where you use a physics engine that expects to resimulate the whole world, and where you are extrapolating many entities that need regular replays, then it's quite likely that you simply won't be able to take this route and will have to pick some other compromise. It's just not the correct simulation model for you.

You might consider some sort of eventually-consistent model, such as applying forces locally that work in the opposite direction to mispredictions to ‘nudge’ the object back closer towards where it should be in future. You have to be careful not to end up with oscillation or instability though.

I must disagree with you on this one. Games like mentioned previously Rocket League resimulate the entire world completely, which contains a bunch of cars and the fully physical ball.

What I went with for now is the same model as rocket league, which is predicting everything including remote cars. When there's misprediction, I snap the remote vehicles to that corrected frame on the physics side and resimulate everything. Then when the resimulation ends, I take the difference between current frame we thought is right, and the one after resimulation of all the frames since correction. Then that difference is gradually smoothed out over time, but only for the visual mesh of the vehicle.
What it allowed me to do is to not use any smoothing for the physics, which would again destroy determinism completely, but I snap it hard according to correction and then smooth out only the visual mesh. With my simulation and ping above 200ms I get amazing results, and the corrections are not even visible.

Very interesting thread, I was following it for a while. Thank you for sharing your (for now) final solution, and thanks to everyone as well.

We have implemented a flying vehicles a few days ago using the exactly same system, just different way that they handle controls of course. It was a great success for them too.

I do see one small thingy in this system, that I don't really know how to fix, however the exact same issue happens in the Rocket League, they just hide it better than me.
Basically, because remote vehicle are predicted, when one of the clients presses and releases acceleration button quickly when the vehicle is idling, you will see that the vehicle in your view will go forward and then snap back. The source of that issue is obvious, we are predicting remote client inputs to a degree, so when they press “W” and we receive that information, we immediately run the vehicle forward as if the remote client kept holding the “W” key, but as it turns out, when we get update soon, he actually released it, so we need to snap his vehicle back.
What I did for that is stronger smoothing with low speeds. This issue is only visible with very low speeds, so we handled these low speeds like a “special case”. It works, and you don't see this issue for up to 120 ping(roughly) beyond 200ms ping it can be already visible clearly, but smoothing helped a lot even with 200ms ping, so it looks a little more like the remote clients moved forward and then backward(because we are rotating the wheels backwards when he snaps back, so it looks like he was just accelerating backwards).
I tested rocket league with pings above 200ms and it does happen for them too. The only difference is that they hide it even better than me somehow. But this is something we will think about polishing later, not really big deal right now.

Fury22 said:

I must disagree with you on this one. Games like mentioned previously Rocket League resimulate the entire world completely, which contains a bunch of cars and the fully physical ball.

True, however, Rocket League has a very simple physics world compared to many games with most objects not being in contact at any given time and the colliders being relatively simple. When your physics step for an object is as simple as “position += velocity” and your collision is a simple distance check (as with a ball) then it's very cheap to replay things.

If you have now found that you do have time for PhysX to resimulate your entire scene, then that's great! That seems like progress relative to earlier in the thread. Sadly it is not an option for most games, especially those with more complex physics with characters, hence why they usually won't attempt to extrapolate remotely-controlled entities if they can avoid it.

Fury22 said:
when one of the clients presses and releases acceleration button quickly when the vehicle is idling, you will see that the vehicle in your view will go forward and then snap back. The source of that issue is obvious

This sounds like some part of the system is not sending important updates quickly enough.