Advertisement

Cloud Architecture/Port problem

Started by November 10, 2014 08:40 PM
17 comments, last by slideside12 10 years ago

multiple matches on the same network interface but in different processes


A "match" is the game match. It requires that all players in the match get told about all other players in the match.

A "network interface" is an abstraction of a physical network card (may be multiple cards for redundancy) and typically has one network address (may have more for various reasons.)

A "process" is an operating system concept that runs an executable, and a context within which a single unit like a file descriptor or socket makes sense (may be inherited across sub-prcesses based on OS.)

A match runs within a process. A process can run more than one match, if it has a way of distinguishing which match incoming packets are intended for.
More than one process CAN bind sockets to the same UDP port, but this is generally a bad idea.

For UDP traffic, when a packet comes in to a network interface for a particular port, any ONE of the sockets that are bound to that port will get that packet.
This means that, if you have multiple processes binding to the same port through different sockets, you have no control over which process gets the data, and because matches are within a process, you can't route the apporpriate packet to the appropriate match.

The solution is typically one of:

1) Use a single process, and single socket receiving on the port, but run many matches within the same process, and use data in the packet to identify the match that a particular incoming packet is meant for.

2) Use multiple processes, but bind each process to a different UDP port. Tell the match players which port to send data to to route it to the right match.

In addition, option 1) probably also want to use multiple threads, to make it scale across CPU cores. Option 2) typically doesn't need that, because each process can be scheduled on a different core.

Ok thanks for the full explanation. If I am to distribute my matches between multiple processes would I have to write a program that acts as an initializer for these processes? So basically that initializer program would be running 24/7 instead of the multiple processes. It handles the initial connection packets and every time a match is full it starts a new process on a new port. However I was also thinking this could be done from within each process itself where after 10 people are in one match the process just launches another instance of itself. Closing the process of course could be done by the process itself when the match is over.

multiple matches on the same network interface but in different processes


A "match" is the game match. It requires that all players in the match get told about all other players in the match.

A "network interface" is an abstraction of a physical network card (may be multiple cards for redundancy) and typically has one network address (may have more for various reasons.)

A "process" is an operating system concept that runs an executable, and a context within which a single unit like a file descriptor or socket makes sense (may be inherited across sub-prcesses based on OS.)

A match runs within a process. A process can run more than one match, if it has a way of distinguishing which match incoming packets are intended for.
More than one process CAN bind sockets to the same UDP port, but this is generally a bad idea.

For UDP traffic, when a packet comes in to a network interface for a particular port, any ONE of the sockets that are bound to that port will get that packet.
This means that, if you have multiple processes binding to the same port through different sockets, you have no control over which process gets the data, and because matches are within a process, you can't route the apporpriate packet to the appropriate match.

The solution is typically one of:

1) Use a single process, and single socket receiving on the port, but run many matches within the same process, and use data in the packet to identify the match that a particular incoming packet is meant for.

2) Use multiple processes, but bind each process to a different UDP port. Tell the match players which port to send data to to route it to the right match.

In addition, option 1) probably also want to use multiple threads, to make it scale across CPU cores. Option 2) typically doesn't need that, because each process can be scheduled on a different core.

Also why can't I just use one process? Each match would be a socket with a different port. So if I had 10 matches going on there would be one process, 10 sockets and each with its own port.

Advertisement

would I have to write a program that acts as an initializer for these processes


It's not a good idea in my experience to write a process that "spawns a copy of itself when needed" -- managing when and how processes are spawned becomes hard (how do you tell all the copies to stop spawning?) Having a coordinator of some sort that spawns a new process would be simpler IMO. That might be as simple as "ssh $hostname /usr/local/bin/myserver -port 1234" run from the main management process.

why can't I just use one process?


Using one process is fine, assuming your code doesn't use any globals. However, using one process with many sockets (and many ports) adds the complexity of managing multiple ports, without the benefit of having each match/socket be somewhat insulated from the failure of another, so it doesn't strike me as a particularly great trade-off. You may find different. I'd just have a single port/socket for a single process. Including a game instance ID in each packet, and/or deriving the intended game instance ID from the remote address you got the packet from, seems better to me.
enum Bool { True, False, FileNotFound };

would I have to write a program that acts as an initializer for these processes


It's not a good idea in my experience to write a process that "spawns a copy of itself when needed" -- managing when and how processes are spawned becomes hard (how do you tell all the copies to stop spawning?) Having a coordinator of some sort that spawns a new process would be simpler IMO. That might be as simple as "ssh $hostname /usr/local/bin/myserver -port 1234" run from the main management process.

why can't I just use one process?


Using one process is fine, assuming your code doesn't use any globals. However, using one process with many sockets (and many ports) adds the complexity of managing multiple ports, without the benefit of having each match/socket be somewhat insulated from the failure of another, so it doesn't strike me as a particularly great trade-off. You may find different. I'd just have a single port/socket for a single process. Including a game instance ID in each packet, and/or deriving the intended game instance ID from the remote address you got the packet from, seems better to me.

Ahh I see. Didn't think about the fact that if I use one process and it crashes all the matches end. One port and socket for each process seems like a much better idea. The game instance ID however; why would that be necessary if both the process/server and all the clients know which port to use? I see how it would work in one big process to identify each match but if all communication is to one port then why use a Game ID? I'm not familiar with the ssh solution you suggested so I might just run a small process that can act as a main management process using C++. However I've only come across ssh a couple of times but if this can be done with a python script on the VM I may do that instead. But I would have to read up on ssh and its uses. I have a much better understanding of everything.

if all communication is to one port then why use a Game ID?


Two reasons:

1) If you don't want to forward the ip:port of each player from the lobby server to the game instance server, you can match up incoming packets from players to the right game session using this ID when the first packet comes in.

2) Some ISPs and/or NAT situations are said to sometimes change the visible IP:port of remote players. If you include the game ID (and a game-specific player ID, equally random-and-hard-to-guess,) then the player session can survive even if the remote IP:port suddenly changes. (This will also happen for roaming devices like phones, laptops being carried around, etc.)
enum Bool { True, False, FileNotFound };


2) Some ISPs and/or NAT situations are said to sometimes change the visible IP:port of remote players. If you include the game ID (and a game-specific player ID, equally random-and-hard-to-guess,) then the player session can survive even if the remote IP:port suddenly changes. (This will also happen for roaming devices like phones, laptops being carried around, etc.)

Because UDP doesn't require an active connection lets say one user's ip changes while in a match. Now when he tries to send a packet to the game instance server it checks if the packets "game ID" matches it own. Is that what you're saying?


1) If you don't want to forward the ip:port of each player from the lobby server to the game instance server, you can match up incoming packets from players to the right game session using this ID when the first packet comes in.

This im not quite understanding. So I can make the lobby the main manager/coordinator process. When a match has been "created" the lobby launches a new game instance server on a new port number and returns that port number to the clients. Each client creates a socket to the dns name (same for the lobby and the match server) with that port. Is the game ID idea used as a verification to allow the game instance server to verify that only those who were matched when the lobby created it will be the only ones allowed in the match? Because you said I can "match up incoming packets from players to the right game session". The only matching I would see needed would be done at the lobby. But once the lobby creates a game it shouldn't be involved again. Communication would only be between the game process and the clients at its port.

Advertisement

Is that what you're saying?


Yes!

once the lobby creates a game it shouldn't be involved again.


Your previous quesiton was this:

if all communication is to one port then why use a Game ID?


Your later example uses a different port per game instance.
enum Bool { True, False, FileNotFound };


Yes!

Great!


Your later example uses a different port per game instance.

Oh ok I think there was some confusion. When I said all communication being to one port I meant for each separate match. I know if all the matches themselves are to one port the game id would be used to identify each match "in" the lobby.

Thank you very much for your help! biggrin.png

Is that what you're saying?


Yes!

once the lobby creates a game it shouldn't be involved again.


Your previous quesiton was this:

if all communication is to one port then why use a Game ID?


Your later example uses a different port per game instance.

Oh one final question. How would I go about including command line arguments when launching a game instance/process from within the main lobby?

Edit: Nevermind I figured it out.

Your help is really appreciated.

This topic is closed to new replies.

Advertisement