Lord have mercy!  

This approach can work depending on your game design imo. It is analogous to splitting up a large world into different shards, with a different server handling each shard, in this case each 'client' in your system is actually a client + server, with those servers communicating to a master server which coordinates between the shards.

The problem as has been mentioned before is in the interactions between these shards. Games of this type can be seen as a continuum between those with no interactions, and those with lots of interaction.

As an extreme example, consider a bank. A bank doesn't care what happens in the real world, that is simulated outside the bank. All it cares about is the transactions that occur into and out of the bank. That is equivalent to this master server. If you have a world e.g. based on trading, where a player only needs to know a list of players in the vicinity and who they can trade with, and there are no physical interactions, this works perfectly, and scales fantastically with lots of players with very little server cost.

The other extreme would be situations in which there are constant interactions between the shards, players sliding off one another, shooting each other etc. In this case, issues like who has authority, lag / synchronization become a big issue, something which is usually dealt with by having an authoritative server. You could for example choose one of the players to be the authoritative server for a local area.

Another option would be to allow players PCs to be authoritative up until they reach proximity with another player, in which case the master server takes control. This might be a way of somewhat decreasing the load on the master server, depending on how much of the time players spent on their own versus next to other players (although might be a nightmare to implement the transitions).

Another big question I would be asking is, given that ultimately this is all an optimization exercise (to shift load off a master server), how much performance are you going to gain using this approach versus simulating at the master server?

Even if you don't simulate on the master server (alternatively read as 'your infrastructure'), if you still seek to communicate via the master server (rather than client -> client), you still (worst case) presumably need to put the transforms of all your simulated objects into packets to send to the master server, it has to unpack / interpret / resend out all of these to other clients in the area. That might be a lot of bandwidth / processing compared to just sending the keyboard input from each client.

If this is indeed the situation you might even find that paradoxically this client simulation approach is *more* not less expensive than server-side simulation. This really requires some profiling and care. Again the degree to which client simulation can be a win depends on the game design, which is why it seems somewhat a chicken and egg situation to me.

If it were me and I was determined to take load off a master server, and you were prepared to trust clients, I think I would go for simply choosing the fastest client with the best connection in an area and making it the local authoritative server. This would be considerably easier to implement / debug than having every client a server, and is already an established solution. Of course you still have the problem of interactions between shards, something which is often 'designed out' (e.g. make one server responsible for one planet / country etc).

I'm not myself aware of others using this 'every client a server approach' in quite the way you suggest (maybe further research would find some), but assuming this was the case, my immediate question would be why? Did they find considerable downsides and opt for other approaches because they found them better?

Sorry Septopus but you kinda let the cat out of the bag and now everyone is trying to fit them all back in.  

I'm not going to speak on behalf of Septopus or what he envisions; however, while reading about this idea of Septopus' I can see an appeal, no central server overhead.  I'm just imagining multiple client's all connecting and this client-server-authoritative system propagating across all those who connect up.
 

5 hours ago, lawnjelly said:

Another option would be to allow players PCs to be authoritative up until they reach proximity with another player, in which case the master server takes control.

For my own attempts to wrap my head around this stuff I like what @lawnjelly suggests. 
What if when two players, A and B, are in close proximity a third player's client-server, C, takes control of authoritative issues for A and B, but never A and C, or B and C?  

I can't speak too much on all this stuff because their is a lot to the world of server infrastructure and protocols and so forth that I am unaware of.  But I do think that if you get this to work, at least the way I half envision, it would be pretty wild stuff!

@Rutin, hehehe.

 

7 hours ago, lawnjelly said:

This approach can work depending on your game design imo. It is analogous to splitting up a large world into different shards, with a different server handling each shard, in this case each 'client' in your system is actually a client + server, with those servers communicating to a master server which coordinates between the shards.

The problem as has been mentioned before is in the interactions between these shards. Games of this type can be seen as a continuum between those with no interactions, and those with lots of interaction.

As an extreme example, consider a bank. A bank doesn't care what happens in the real world, that is simulated outside the bank. All it cares about is the transactions that occur into and out of the bank. That is equivalent to this master server. If you have a world e.g. based on trading, where a player only needs to know a list of players in the vicinity and who they can trade with, and there are no physical interactions, this works perfectly, and scales fantastically with lots of players with very little server cost.

I can see why you see it as a client+server type system..  But really, it's not quite that.  Here's how it works.

Player inputs from keyboard/mouse.  That input is sent through their local physics simulation instead of to the server first and the resulting final frame coordinates are sent to the server for processing/broadcasting.  This is completely analogous to the client side prediction used in centralized server systems, except the server doesn't get the coords until the client has already "predicted" the outcome.  The resultant network objects are then kinematically interpolated into the other "simulations" that they interact with, using only collider physics as all movement is interpolated for these objects.  Also in an analogous fashion to how this is handled in single simulation systems.  My environment has a single game-wide and stable coordinate system, no level loading, no significant variances in any aspect of the positioning of objects from one client to the other.  So far in all my multi-player tests, the only discrepancies I've been able to note are a slight delay in the movement of network characters due to the network lag/interpolation.  Granted, I've yet to test npcs with multiple clients(still writing the code) and anything with "fast physics" like bullets/etc..  Truth is, I may not implement a typical weapons system in this game.  I may use something like energy weapons with a built in spread effect or EMP / explosives based tech that has a radius of damage.

7 hours ago, lawnjelly said:

The other extreme would be situations in which there are constant interactions between the shards, players sliding off one another, shooting each other etc. In this case, issues like who has authority, lag / synchronization become a big issue, something which is usually dealt with by having an authoritative server. You could for example choose one of the players to be the authoritative server for a local area.

Another option would be to allow players PCs to be authoritative up until they reach proximity with another player, in which case the master server takes control. This might be a way of somewhat decreasing the load on the master server, depending on how much of the time players spent on their own versus next to other players (although might be a nightmare to implement the transitions).

Another big question I would be asking is, given that ultimately this is all an optimization exercise (to shift load off a master server), how much performance are you going to gain using this approach versus simulating at the master server?

It's actually an exercise in Distribution, not optimization.  The goal is an optimized system, but only one that performs at least as well as a centralized system(if it's better WOOO!).

7 hours ago, lawnjelly said:

Even if you don't simulate on the master server (alternatively read as 'your infrastructure'), if you still seek to communicate via the master server (rather than client -> client), you still (worst case) presumably need to put the transforms of all your simulated objects into packets to send to the master server, it has to unpack / interpret / resend out all of these to other clients in the area. That might be a lot of bandwidth / processing compared to just sending the keyboard input from each client.

As stated above, this bit works a little differently than that.  The servers only receive periodic(very frequent still) updates as to the state of a client held object's physics outcome.  These coordinates are not used to broadcast the object coords to other clients, they are used for authoritative verification purposes.  Live transform data is shipped out to the message routing pub/sub system where each client who is subscribed to the feeds that represent their view of the world will pick up the transform data and interpolate it into it's game view.  The authoritative servers, having a direct connection to each client that bypasses the pub/sub system can then "block" and issue a "reconcile" order for any potential paradox they can detect forming.

That's mostly how the NPCs and other objects work anyhow, player characters are not using the pub/sub system as yet..  So they update the authoritative servers via UDP(post phyics sim) and the authoritative server rebroadcasts that transform data out to the rest of the concerned world.  This operates almost the same as every other system, except the processing of the physics doesn't happen at the server and thusly, the rebroadcasting of the data happens much faster.  I may fold the player characters into the pub/sub system too, but I'm still writing all that code and I'm using the NPCs as the proof of concept for the pub/sub mechanism.

7 hours ago, lawnjelly said:

If this is indeed the situation you might even find that paradoxically this client simulation approach is *more* not less expensive than server-side simulation. This really requires some profiling and care. Again the degree to which client simulation can be a win depends on the game design, which is why it seems somewhat a chicken and egg situation to me.

If it were me and I was determined to take load off a master server, and you were prepared to trust clients, I think I would go for simply choosing the fastest client with the best connection in an area and making it the local authoritative server. This would be considerably easier to implement / debug than having every client a server, and is already an established solution. Of course you still have the problem of interactions between shards, something which is often 'designed out' (e.g. make one server responsible for one planet / country etc).

I'm not myself aware of others using this 'every client a server approach' in quite the way you suggest (maybe further research would find some), but assuming this was the case, my immediate question would be why? Did they find considerable downsides and opt for other approaches because they found them better?

Again, it's an exercise in Distributing the roles of a single server in a game system.  Primarily the physics processing role, as that's the only aspect that's really hard to distribute effectively.  I'm trying to bring the authority as close to the client as possible without giving all the control to the client.  The ultimate goal is lowered lag times, or at least a better play experience regardless of lag, for MORE players on the entire network.

@Awoken unfortunately, I find cats never stay in bags long.  Even when they get in there on their own. haha

@Rutin LOL!! 

36 minutes ago, Septopus said:

I can see why you see it as a client+server type system..  But really, it's not quite that.  Here's how it works.

Player inputs from keyboard/mouse.  That input is sent through their local physics simulation instead of to the server first and the resulting final frame coordinates are sent to the server for processing/broadcasting.  This is completely analogous to the client side prediction used in centralized server systems, except the server doesn't get the coords until the client has already "predicted" the outcome.

Sorry I wasn't too clear .. it is common to run client + server on the same machine in a game designed for multiplayer, even to implement single player. Thus your client scenario is equivalent to a client + local server (with the local server being equivalent of a shard in a bigger world). If you want to mix your client + simulation code that is doable, but it may make the option more difficult to run the client simulation on another machine if you decide to change this.

It does sound a complex arrangement, it will be interesting to hear how you get on dealing with lag / synchronisation / authority issues in practice, you are braver than I!

9 minutes ago, lawnjelly said:

Sorry I wasn't too clear .. it is common to run client + server on the same machine in a game designed for multiplayer, even to implement single player. Thus your client scenario is equivalent to a client + local server (with the local server being equivalent of a shard in a bigger world). If you want to mix your client + simulation code that is doable, but it may make the option more difficult to run the client simulation on another machine if you decide to change this.

It does sound a complex arrangement, it will be interesting to hear how you get on dealing with lag / synchronisation / authority issues in practice, you are braver than I!

Yeah, maybe I'm arguing semantics there a bit?

I guess I'm not catching your contrast of "client + local server VS client + simulaton"..

Maybe because you're still thinking of the client as having/controlling a "shard" of the world.  I think that's still a misconception.  The client only has a view of the one single world shard.  The client uses it's physics engine as the motor force for movement and collision.  The physics of the client never directly propagate beyond the client's immediate view, only their transform data does that.

and I'm still wrapping my head around all the ins and outs of the concept myself so sometimes I explain very poorly..

In a way you could say that every single physics simulation is an individual concurrently running simulation of the game state as authorized by the servers and viewed from that client's angle.

Because of this, it will be hard to implement remote viewing, or replay to any exacting measures, since there's never an actual authoritative "physics" game state.  Replay can happen, but it will require a full interpolation based replay of the recorded authorized data, and as a result will lose a significant amount of data resolution.

I remember one guy working on a top down GTA alike game using physics engine, and i suggested him to use weak forces and torques to sync bodies across multiple clients over time (so maybe just a more advanced version of interpolating or averaging). He said it worked well and the external force was not noticeable to the physics driven game.

I have no experience with networking at all and i do not know if this is common practice or has a proper name. And i would expect problems e.g. if a body goes on both sides of a wall on two different clients (it would end up sticking to the wall like with a rubberband attached  )

However it sounds like an uncomplicated solution for physics alone, leaving logic like health points as a different problem where lag and dictating ownership is more acceptable.

... not sure how much related to the things discussed here this can be.

6 hours ago, JoeJ said:

I remember one guy working on a top down GTA alike game using physics engine, and i suggested him to use weak forces and torques to sync bodies across multiple clients over time (so maybe just a more advanced version of interpolating or averaging). He said it worked well and the external force was not noticeable to the physics driven game.

Hmm, that is interesting.  I may explore that a bit too, thanks!

1 hour ago, Septopus said:

I may explore that a bit too

The threat was this: http://newtondynamics.com/forum/viewtopic.php?f=9&t=9224

It has comparison videos and math.

1 hour ago, Septopus said:

I may explore that a bit too

The threat was this: http://newtondynamics.com/forum/viewtopic.php?f=9&t=9224

It has comparison videos and math.

Very cool, thanks for the link!