Rollbacks and Simulation Replay - Performance

Fury22 · 2023-07-30T03:15:31

Hi everyone! Despite being a successful games programmer for a few years now, this is the first time I decided to post here on forum ? So welcome!I am building a multiplayer game right now that involves cars, that are simulated physically. So I have been doing various networking for years now, and I am very familiar with the client/server architecture, client side prediction, lag compensation and server reconciliation techniques. However I mostly worked on netcode for shooter games, which was a bit simpler than the physics simulation I am dealing with right now.My technologyI am using Unreal Engine, but that is completely irrelevant as I have integrated separate physics engine(the newest NVidia PhysX) to be able to manually tick the physics engine in order to replay physics and deal generally with the fixed time step. So I have complete control over the physics for my vehicles and the entire scene, and Unreal doesn't interfere, I made sure of it.My solution allows me to change time steps at edit time, so I am able to test various fixed time steps without any trouble.My networking solutionI am using standard networking solution that has been introduced during multiple GDC talks such as Overwatch, Rocket League, Halo and Glenn Fiedler's networked physics talk.So, my clients are always running ahead of the server by half RTT + 1 buffer frame. Clients are using upstream throttle in order to simulate faster or slower, which is dictated by the server, to always have some inputs buffer on the server side. This solution is very robust as I have shipped multiple FPP and TPP shooter games without any trouble. The clients only send a list of inputs to the server, from oldest not-yet-acknowledged input, to the current one, so packet loss is covered very well(standard technique).For the characters it's all simple, local client is using its own inputs to predict local client and buffers the states and inputs locally, so when server returns state, it checks if correction is needed, and if so, it replays all the frames from the corrected one up until current one to be back on the lockstep. It all works perfectly, I wanted you to be aware I am very familiar with this networking model.Now to the point…The regular player characters can be rolled back to the correction frame and replayed extremely easily, because they do not have any physics, they are kinematic beings and therefore their “physics” replays are very, very cheap.I started implemented my vehicle system, which is deterministic enough(except for the floating point errors, but they are well covered using the corrections, so this can be completely skipped). Each input for the vehicle on all clients and server gives exactly the same results, which is very big win for me. However….My game is heavily focused on cars collisions, and that's where the trouble start!Predicting everything(like Rocket League)Simulation running at 60hzIn this solution I am predicting local player, which is easy, due to the deterministic nature of the vehicle and physics system I have. There are never ever mispredictions, even when hitting some static objects around.However, OTHER CARS are not on the same timeline for me as I am. I used the Rocket League solution, where I would predict remote clients just as I predict mine. In this case when I get information about the remote client from the past, I check if it needs to be corrected. If so, I do the replay from that corrected frame to the current one. Standard technique and it works very well. There are some mispredictions pretty often, because I cannot predict remote client's input, so all I have is their last used input. If they have changed the direction during that time, I have no way to know that, but that's okay, the correction and replay covers that well enough, so I don't really see that.This technique seems perfect, but… performance. I measured my simulation and 1 physics frame takes usually around 0.5ms - 1ms time, which is very quick, really nice. However. Let's say I am on the frame 100, but because I am high ping client(let's say 200ms ping), I may receive information about remote clients from around 200 or more milliseconds ago, which with 60Hz simulation translates to around 13 frames. This is where the trouble starts. If my simulation usually takes around 1ms, I need to replay the simulation from 13 frames ago, up to current one, which is 13ms. This IS SIGNIFICANT. Considering, that the game rendering at 60FPS(16ms), this is over 80% of the time to be added! Of course, we can try to budget the game in a way that it is able to run at 120fps, when we would have enough budget to keep it there, but I am wondering if I am missing something? Or is it really the way it needs to be? Considering how remote clients go out of sync as soon as they change their inputs, this can very often simulate 13 frames of physics during one fixed time step tick. I do not think I can squeeze the physics to simulate faster than that 0.5 to 1ms, this is pretty good as it is.So do you think I am missing something here? Or is this they way to go, and we should consider optimizing other parts of the game to account for that extra 13ms? Thanks for any responses!

Networking and Multiplayer Programming Netcode Rollbacks networking simulation Physics

Started by Fury22 September 25, 2022 07:16 PM

23 comments, last by Tom Sloper 1 year, 4 months ago

hplus0603

11,938

November 22, 2022 05:10 PM

Kylotan said:
some part of the system is not sending important updates quickly enough

It's true, you should aim for the highest possible packet rate for control inputs for players who are visible. 60 Hz packets are totally possible on most current internet connections.

That being said – latency is latency. If you forward extrapolate, you will absolutely overshoot when players make drastic control changes. The options are to either try smoothing the overshoot, or to not forward extrapolate. Which variant works best, depends on the specific game.

enum Bool { True, False, FileNotFound };

Fury22

Author

November 24, 2022 12:26 AM

Right. Well, for me only the prediction was the option. I mean, I couldn't figure out how to do it otherwise.

My physics is very simple. All vehicles are just boxes, I purposely keep physics as simple as possible, which gives me very good deterministic results.
I could only get it to work using prediction. In my game vehicles will be colliding with each other quite often. I don't see how I could handle it any other way. If I didn't predict remote vehicles, local player would see the remote clients from the past, which for the vehicles could be like a few meters behind and if I locally wanted to collide with the remote client, I would be colliding with his version from the past, which gave me really, really bad results. Especially for that guy I've hit. He was sure to be far ahead of me and safe, while on my screen I hit him, and then what? I tried with lag compensation on the server side like I do for shooting, but this was the most horrible experience ever. You could be pushed around by the vehicle you see behind you.

Kurdur

105

July 29, 2023 10:33 PM

Stumbled across this and it was a really informative discussion, thanks! Thought I’d share this for future readers like myself.

I tested rocket league with pings above 200ms and it does happen for them too. The only difference is that they hide it even better than me somehow.

They do this by “by decaying a player’s input over time based on how old that input is. For example, the first frame you’re predicting a remote player you apply 100% of their movement input, the second frame two-thirds, the third frame one-third, and by the fourth frame you don’t consider the remote player’s movement input at all. The intuition here is that undershooting motion and then catching up when remote input is received tends to look a lot better than overshooting and then rubber-banding backwards.”

This is from the April 2023 SnapNet article on predict-rollback part 2. Hope it’s helpful!