Hey, my first post here. I want to ask for help in understanding something. So lets say you are working on a simple 2D desktop application. A space shooter. The application is written in C++ or C#. No framework or game engine is being used. I want to implement a system where the user can select multiplayer from a menu and after loading they will be put into a match with about 9 other players. These players would obviously be other players who were searching for a game at the time. Once all the players are connected to the match, they will play a simple multiplayer game match on a tilemap. After 3-4 rounds the person who won the most rounds wins. Very simple.

Now I understand the basics of getting this done. Create a local server in C++ and use packets to send information from one player to the others. This information would include the players position, his rotation and when he shoots (a spacehip game for this example). Apply this for each player on each frame (this is a real time game). Ive read a lot so I understand the basics of UDP/TCP and the structure of packets. However I do not understand everything fully.

1. Having alot of experience with Google App Engine and Python I wanted to know how I could use their channel api to accomplish this. Ive seen games such as Realm of the Mad God accomplish this and its been cited that it uses app engine. Many people have said their Channel API is not a good fit for real time multiplayer games due to latency. Now not only is this game real time but it allows for 85 players for a single battle and is browser based. Now if I am trying to accomplish 8-10 players, desktop based and no mmorpg functionality (open world, transitioning between worlds, online world state persistence, inventory, etc.) I don't see why I couldn't accomplish this using their api. Another problem I have is that their api says messages are sent between the backend and a javascript client. How would I accomplish this since I am programming for the desktop?

2. I have heard of node.js and socket.io as being nice solutions to my problem but once again they are browser based. How would I implement something like that for desktop?

3. Smartfox seems like a good solution but I don't really understand how it full works. Honestly though this looks like the best solution for what I want to achieve if I understood it more. Not understood how to use it but what it does/what role it plays. Do you use it to write a server/client, upload to a hosting service and then run it on there?

4. In the event I use a service like Amazon EC2 and write a simple c++ server and upload it I don't quite understand how the local implementation translates to working on the web. I know for the local server/client model I can just use a local ip and the same port. But lets say I have 1000 players and each match can contain 10 players. Would I just create a new port for each match? I haven't looked much into EC2 but I am assuming you would get an IP address for a server and then each port could act as a match. So the first match would be on ip_to_server:0000 then the next on ip_to_server:0001 and so on and so on.

5. What would be the security concerns for using something like a simple c++ server/client uploaded to a cloud hosting service? I know from my HTTP experience that the main security is in encrypting/hashing the data sent through HTTP requests. Now obviously I cannot do that with packets as that would cause latency and lag due to the increased size of the messages. And also the information being sent isn't private so I don't really see where security comes into play here. Would the security be the cloud service I use?

6. If a service like EC2 is used where do the costs come from? Is it based on the amount of messages sent between the sockets?

I guess the best case scenario I am looking at is using the channels api and just sending messages from the desktop application to a backend that acts as a a server and worst case I would write a simple c++ server, upload it to a service like EC2 and use a port for each match? Once again this is very simple.

Thank you in advance.

Google App Engine, socket.io, and SmartFox are all "nice" in the sense that they work with a standard HTTP-based web browser network stack.

If you're building a desktop/mobile app on top of native code, though, you are not limited by what the browser can support. You can use native networking libraries like RakNet (C++,) ENet (C) or Lidgren (C#.) The benefit is that, once the players are matched up, the server doesn't need to worry about the network traffic; one of the players will be "host" and the others are treating that player as "server."

You will need one server still. The role of that server is to make the players find each other. If you want matchmaking to be "automatic," then you just let the player "connect," and what actually happens is:

1) player A ocnnects to the server
2) if server finds that there is an active session looking for players, it tells player A to connect to that session; done
3) if not, it tells player A to start hosting a session, and remembers player A
4) when player B comes in, it's told to connect to player A
5) when the server has forwarded 10 players to the same session, or the hosting player's machine says "I'm full," the server stops forwarding new players to that session; the next player to connect will be told to host another session

This mechanism can be improved in many ways -- support specific filters ("I only want to play in co-op mode games") and choosing the best host ("when I have 10 players, I pick the one with the fastest ping and tell it to host, and tell the others to connect") as well as using this server as NAT introducer.

Google App Engine, socket.io, and SmartFox are all "nice" in the sense that they work with a standard HTTP-based web browser network stack.

If you're building a desktop/mobile app on top of native code, though, you are not limited by what the browser can support. You can use native networking libraries like RakNet (C++,) ENet (C) or Lidgren (C#.) The benefit is that, once the players are matched up, the server doesn't need to worry about the network traffic; one of the players will be "host" and the others are treating that player as "server."

You will need one server still. The role of that server is to make the players find each other. If you want matchmaking to be "automatic," then you just let the player "connect," and what actually happens is:

1) player A ocnnects to the server
2) if server finds that there is an active session looking for players, it tells player A to connect to that session; done
3) if not, it tells player A to start hosting a session, and remembers player A
4) when player B comes in, it's told to connect to player A
5) when the server has forwarded 10 players to the same session, or the hosting player's machine says "I'm full," the server stops forwarding new players to that session; the next player to connect will be told to host another session

This mechanism can be improved in many ways -- support specific filters ("I only want to play in co-op mode games") and choosing the best host ("when I have 10 players, I pick the one with the fastest ping and tell it to host, and tell the others to connect") as well as using this server as NAT introducer.

Large Edit:

Thanks for your fast response.

Initially I had a long response with a lot of questions. I sort of answered some of them myself after searching for a while but I still want to clear everything up so I have condensed them into the following questions:

1. I thought your suggestion was a peer to peer model but i've realized it is actually client/server model. So in this model I would program two servers and a client. One server (master server - used for matchmaking) I deploy to an actual online hosting service and one (host server - used to act as the server for multiple clients) is included inside the game code. Client functionality is also included in the game code. It is just what ip address to connect the socket to and how to send/receive packets to/from that ip address. So for an example situation. Once I have 10 players (for c++ = strings stored in a vector with each id attached to its ip address) I would just have the host launch the "host server" on his machine. This server would start listening for connections to his ip address. Then on each of the clients machines the master server would send them the ip address of this host. Each client would connects through a socket to the host's ip address. After that I just handle the gameplay. Every time a client moves for example I would send a packet to the host. The host would then update that client's movement on his own machine and send a packet to all the other clients. Then the clients would update. This seems like a very powerful and useful model. I think I've come across it before but now I finally understand it after your breakdown why its useful and how it can cut down costs for example. I believe this is how most online FPS work.

2. In your host/clients model how would you deal with something like cheating? Because technically the host can send false data about his movements to the other clients as well as manipulate the packets he receives before sending them to the clients.

3. If I wanted to implement things like leaderboards and user accounts I know how to implement this myself on the database/storage side. But would relaying this information be as simple as sending the final game data (just account ids w/ scores) to a backend with a https request once the game is done? Or would it be better to create a socket connection and send the data? If the host disconnects though I assume the match would be lost.

4. Ok so I have a better understanding of services like Amazon EC2. So if I choose them for example I really wouldn't need the multiple ports as I was saying earlier. Just one instance and its ip address basically. I would install the exe of the master server on it and run it 24/7. Since each instance is like a physical running machine I guess I could use the "get local ip address" function in whatever library I choose and start listening for connections to that ip address? If I wanted to launch multiple instances for various regions how would the game/player know which instance to connect to? I don't want to create an instance whose only job is just to send the ip address of the correct master server for the player to connect to (seems like a waste of resources).

5. In the case I want to still use the server/client model where I the actual cloud server acts as the server what would be the best way to do this with EC2 or Google Compute Engine? I guess I could have an instance/server for various regions but clearly not for every small match. So now in this case would I use a different port for each match on thats on the same server?

6. Exactly what maintenance would be needed in these situations? The only thing I could think of is when you see an instance being overworked for example just create another one. But then that would mean I would have to update the game that is on the clients computer I'm assuming. I don't really understand how patches work with already created executables but this probably isn't the section for asking this. Same question for security. The only thing I could see as a possible risk is a DDOS attack. Every other security related issue would be handled by the cloud provider.

P.S. You mentioned Raknet and I had said I looked into it. There are pricing models on the website but Im sure you are aware Oculus purchased it and it was made open source. Wouldn't this make it free no matter how many platforms its deployed to or its price.

Thank you in advance. Everything is coming together now and I feel confident in implementing this.

2. cheating
There are many options there. One is to realize that your game is small and cheating isn't yet a problem :-)
Larger games, like Counter-Strike or Team Fortress, deal with this by using systems like Valve Anti-Cheat. (Is using Steamworks an option for you?)
Other options include collecting statistics about players/installs, and score them based on discrepancies detected and reported by each client. Note that a single client that detects more discrepancies than others would likely be trying to cheat the system!
You could also pay for each game server, and let players play on servers you have hosted. This makes the economics of scaling your game harder, altough if you provide "play on non-cheating servers" as an up-sell premium feature, that might work.
Also, certain cheats (wall hacks, aimbots, etc) aren't detectable over the network; VAC does client-side scanning to find cheats like that.

3. leaderboards
Yes, you can have the game participants post results. Again, this may open you up a bit for cheating, but if you have each client and the server all post the same data, and verify it, the risk is smaller. (Three friends could still collaborate, create a match, and the use a cheat script to post inflated scores.)

4. EC2 addresses/ports
EC2 local addresses are not public. You will be assigned a public IP address, but that's not the address of the machine -- Amazon's firewalls will do port forwarding for you.
However, that doesn't matter -- servers typically bind to the address 0,0,0,0 (IPADDR_ANY) which means "listen on all local interfaces," and then to a particular port; you'll have your publicly assigned IP (52.153.254.88 or whtatever) open up that port in the Amazon control panel, and your server will see the incoming connection.
For the players to connect to your server, you need to arrange for a domain name ("master.your-super-duper-game-domain.com" say) that maps to the IP address that's public for your server.

5. cloud servers for games
Virtualized cloud servers have significant scheduling jitter. At any point (and frequently,) your server instance will "stop" for between 1 and 1500 milliseconds, before it starts running again. The smaller instances are worse than the larger instances, but it also happens to the larger ones. If your game needs to run one tick every 16.6 milliseconds, game quality will not be good on virtual servers.
The solution is to use "bare metal" "self-managed server" hardware, which you can get through places like serverbeach.com, or interserver.net, or others. Make sure it's actually "bare metal" and not in any way "virtualized."
For larger game companies, it ends up being cheaper to start leasing space in some co-location facility, screw in rack-mount machines, and pay some ISP for transit, and running your own data center.
If your game is not a high-precision action game, virtualized servers can work just fine, and makes life easier.

6. scaling
DNS can be set to map to more than one IP address for a given name. Thus, if you have four servers with public IPs, you can make your clients roughly load balance across them by returning all four IPs for the one name. The server returns them in a randomized order, and the client typically picks the first of the list.
Also, most network load balancers/routers (including Amazon Elastic Load Balancer service) can "pick one" of the machines on the inside for each connection to the same IP from the outside. You can then have N servers on the inside, all responding to the same IP address from the outside.
Note: Two players that connect to different physical hardware will then not be able to "see" each other, unless you build a connected mesh server back-end of some sort. Chances are, now is not the time to worry about that for your game -- if you can fill up a single machine with matchmaking requests, you'll be doing tremendously well and can afford to throw money at the problem at that point :-)

I feel confident in implementing this.

"What could possibly go wrong?" :-)

The six stages of a project:

1. Irrational Enthusiasm
2. Disillusionment
3. Panic
4. Search for the guilty
5. Puninshing of the innocent
6. Praise for the non-participants

You are in stage 1.

2. cheating
There are many options there. One is to realize that your game is small and cheating isn't yet a problem :-)
Larger games, like Counter-Strike or Team Fortress, deal with this by using systems like Valve Anti-Cheat. (Is using Steamworks an option for you?)
Other options include collecting statistics about players/installs, and score them based on discrepancies detected and reported by each client. Note that a single client that detects more discrepancies than others would likely be trying to cheat the system!
You could also pay for each game server, and let players play on servers you have hosted. This makes the economics of scaling your game harder, altough if you provide "play on non-cheating servers" as an up-sell premium feature, that might work.
Also, certain cheats (wall hacks, aimbots, etc) aren't detectable over the network; VAC does client-side scanning to find cheats like that.

3. leaderboards
Yes, you can have the game participants post results. Again, this may open you up a bit for cheating, but if you have each client and the server all post the same data, and verify it, the risk is smaller. (Three friends could still collaborate, create a match, and the use a cheat script to post inflated scores.)

4. EC2 addresses/ports
EC2 local addresses are not public. You will be assigned a public IP address, but that's not the address of the machine -- Amazon's firewalls will do port forwarding for you.
However, that doesn't matter -- servers typically bind to the address 0,0,0,0 (IPADDR_ANY) which means "listen on all local interfaces," and then to a particular port; you'll have your publicly assigned IP (52.153.254.88 or whtatever) open up that port in the Amazon control panel, and your server will see the incoming connection.
For the players to connect to your server, you need to arrange for a domain name ("master.your-super-duper-game-domain.com" say) that maps to the IP address that's public for your server.

5. cloud servers for games
Virtualized cloud servers have significant scheduling jitter. At any point (and frequently,) your server instance will "stop" for between 1 and 1500 milliseconds, before it starts running again. The smaller instances are worse than the larger instances, but it also happens to the larger ones. If your game needs to run one tick every 16.6 milliseconds, game quality will not be good on virtual servers.
The solution is to use "bare metal" "self-managed server" hardware, which you can get through places like serverbeach.com, or interserver.net, or others. Make sure it's actually "bare metal" and not in any way "virtualized."
For larger game companies, it ends up being cheaper to start leasing space in some co-location facility, screw in rack-mount machines, and pay some ISP for transit, and running your own data center.
If your game is not a high-precision action game, virtualized servers can work just fine, and makes life easier.

6. scaling
DNS can be set to map to more than one IP address for a given name. Thus, if you have four servers with public IPs, you can make your clients roughly load balance across them by returning all four IPs for the one name. The server returns them in a randomized order, and the client typically picks the first of the list.
Also, most network load balancers/routers (including Amazon Elastic Load Balancer service) can "pick one" of the machines on the inside for each connection to the same IP from the outside. You can then have N servers on the inside, all responding to the same IP address from the outside.
Note: Two players that connect to different physical hardware will then not be able to "see" each other, unless you build a connected mesh server back-end of some sort. Chances are, now is not the time to worry about that for your game -- if you can fill up a single machine with matchmaking requests, you'll be doing tremendously well and can afford to throw money at the problem at that point :-)

I feel confident in implementing this.

"What could possibly go wrong?" :-)

The six stages of a project:

1. Irrational Enthusiasm
2. Disillusionment
3. Panic
4. Search for the guilty
5. Puninshing of the innocent
6. Praise for the non-participants

You are in stage 1.

1. You didn't respond at all to number 1 so I'll assume everything I said there is correct as far as implementation goes.

2. As for the non-cheating upsell idea that is good and having users play on the server I host myself to prevent cheating is why I was still asking about that. Ive seen VAC and Steamworks a couple of times and they do seem like a possible options but I have to look more into it. Is VAC essentially a library/sdk that you install into your game and sent parameters for it to detect? I assume this would have to be used with Steamworks for it to properly work.

4. For the EC2 ip/port issue after reading I know how I could do it manually but I was just saying in the case the ip address changes or something the server is getting the address itself but from what you've said it seems you cannot get the public ip address from a VM running on an instance. But elastic ip addresses seem to allow for an ip address that won't change so I guess if anything ever goes wrong I can just update the ip address manually and update the server file in the instance. This is strange though because when running locally you can get the public ip address so why not on a VM?

5. Yes I understand the virtual server model but what I am asking here is in relation to multiple matches on each server. I know for large scale MMORPG games one server holds all the player so I guess they all use the same port. Even if I were to reach a large amount of users it would still just be matches so I wouldn't have that open world type multiplayer. Now obviously each match would have a set of sockets that the server sends packets to. But what I want to know is that when I create a new match do I make bind the sockets with one port for that match. Then add a new port for every new match. Or does every socket connect to the same port and then I just send it accordingly?

6. So for your load balancing you're basically saying there would still be one server running for every region but it would have the resource of multiple machines. Each machine having the same ip address. But wouldn't that mean each machine running the server themselves? Don't really see the advantage of randomly selecting IP addresses based on one DNS name. If I had multiple servers I would rather have it based on region. Each ip address would just be for one server and each server would run on only one machine. This assures enough power for one server and eliminates the possibility of the "not on the same hardware but same IP" problem. For the more power situation I could just start new instances with more power. Also would the not seeing each other problem apply for both the "server based server/client" model and the "host based server/client" model? Could you tell me the advantage to what you suggested because I don't really see it.

Thanks a lot once again. Also I would say i'm more around stage 2

elastic ip addresses seem to allow for an ip address that won't change so I guess if anything ever goes wrong I can just update the ip address manually and update the server file in the instance

Do not hard code an IP address in an executable. This is what DNS is for.

when running locally you can get the public ip address so why not on a VM

No you cannot. The local address is typically 192.168.0.2 or 10.0.0.5 or whatever -- some private, non-routable address that your firewall/NAT router hands out to you. Getting the public IP address of your firewall/router is not possible through regular API means.

5.

If you host multiple matches on the same network interface but in different processes, then you want a different port per match. If you host all the games in the same process (using time slicing or threads or whatever) then you CAN use the same port/socket, and let each user packet include an identifier saying "I play in game session X." Make that ID hard to guess (random) to avoid hijacking.

6.

Scaling by region is fine, for as long as that works. You might split by continent, then country, then state. However, what do you do when you have more players in NYC than a single server can host? That's when multiple physical servers in the same data center are required. I suggested two different ways you can acoomplish that; which one you choose is up to you. (And chances are, your game won't grow to be big enough that it needs that :-)

Do not hard code an IP address in an executable. This is what DNS is for.

So I can instead hardcode the dns into the executable? How is this different if they both point to the same thing?

No you cannot. The local address is typically 192.168.0.2 or 10.0.0.5 or whatever -- some private, non-routable address that your firewall/NAT router hands out to you. Getting the public IP address of your firewall/router is not possible through regular API means.

Oh yeah because of NAT. I guess I can include a nat punch through system if I go with the host/client model.

If you host multiple matches on the same network interface but in different processes, then you want a different port per match. If you host all the games in the same process (using time slicing or threads or whatever) then you CAN use the same port/socket, and let each user packet include an identifier saying "I play in game session X." Make that ID hard to guess (random) to avoid hijacking.

The random id and such I am familiar with. Its like authentication when working with web services/HTTP requests. But the terminology you used for the first part was a bit confusing. When you say "multiple matches on the same network interface but in different processes" what do you mean?

Scaling by region is fine, for as long as that works. You might split by continent, then country, then state. However, what do you do when you have more players in NYC than a single server can host? That's when multiple physical servers in the same data center are required. I suggested two different ways you can acoomplish that; which one you choose is up to you. (And chances are, your game won't grow to be big enough that it needs that :-)

Most games i've seen don't go by state though. Thats pretty specific. Its usually by regions of continents/countries depending on part of the world. But yes I see what you are saying. However it seems the general notion here is that most people's games wont get far. Guess we'll have to wait and see then.

Oh yeah because of NAT. I guess I can include a nat punch through system if I go with the host/client model.

No. Your servers local IP address is completely useless.

Clients will only be able to come in through the servers public IP.

So I can instead hardcode the dns into the executable? How is this different if they both point to the same thing?

Because you can change what the DNS name points to on the fly. To change a hard coded IP requires that you contact all 200 million players and have them redownload your new binary. Your IP may change several times throughout the year if you upgrade servers. Your DNS name itself doesn't have to change, you just change what it points to.

The random id and such I am familiar with. Its like authentication when working with web services/HTTP requests. But the terminology you used for the first part was a bit confusing. When you say "multiple matches on the same network interface but in different processes" what do you mean?

He means you have multiple copies of myserver.exe running. Each will all have the same IP (interface), but the OS will not allow them to use the same port.

A process is a term for a running program in lame speak. Processes contain one or many threads but each process has its own memory space.

Each myserver.exe copy would be hosting a different match.

No. Your servers local IP address is completely useless.
Clients will only be able to come in through the servers public IP.

Gotcha. So if I use the "user as a host" model, through nat punchthrough, I would set up communication between the users through the master servers public ip (dns name now that I understand). Then send the "hosts" public ip (stored on the master server) to each client. I know theres more with having the host send a packet to each client first to establish a connection and such but im already aware of how to implement that. I would just use this NAT punchthrough system for every match then because I can't tell who is behind a firewall or who has a nat router.

Because you can change what the DNS name points to on the fly. To change a hard coded IP requires that you contact all 200 million players and have them redownload your new binary. Your IP may change several times throughout the year if you upgrade servers. Your DNS name itself doesn't have to change, you just change what it points to.

Ok thanks I finally understand it now.

He means you have multiple copies of myserver.exe running. Each will all have the same IP (interface), but the OS will not allow them to use the same port.
A process is a term for a running program in lame speak. Processes contain one or many threads but each process has its own memory space.
Each myserver.exe copy would be hosting a different match.

Oh ok like computer processes. Well then in your opinion what would be the most reliable/fastest way between the two choices? I would assume the one port per process implementation especially since each process would have its own memory space which is less likely to crash than trying to use one process for thousands/millions of socket connections. But this would of course have to be automatic. So I could write a program which would be ran on each VM. This program would act like an orchestrator. When one myserver.exe has had enough players for that match and a new match is being created the orchestrator program would launch a new process. The quiting code I could have inside the myserver.exe for when there are no players in a match. So the actual "process" that would be running 24/7 on each instance would be this orchestrator.exe file.

Thank you very much. Your responses are very helpful and so is this forum.

multiple matches on the same network interface but in different processes

A "match" is the game match. It requires that all players in the match get told about all other players in the match.

A "network interface" is an abstraction of a physical network card (may be multiple cards for redundancy) and typically has one network address (may have more for various reasons.)

A "process" is an operating system concept that runs an executable, and a context within which a single unit like a file descriptor or socket makes sense (may be inherited across sub-prcesses based on OS.)

A match runs within a process. A process can run more than one match, if it has a way of distinguishing which match incoming packets are intended for.
More than one process CAN bind sockets to the same UDP port, but this is generally a bad idea.

For UDP traffic, when a packet comes in to a network interface for a particular port, any ONE of the sockets that are bound to that port will get that packet.
This means that, if you have multiple processes binding to the same port through different sockets, you have no control over which process gets the data, and because matches are within a process, you can't route the apporpriate packet to the appropriate match.

The solution is typically one of:

1) Use a single process, and single socket receiving on the port, but run many matches within the same process, and use data in the packet to identify the match that a particular incoming packet is meant for.

2) Use multiple processes, but bind each process to a different UDP port. Tell the match players which port to send data to to route it to the right match.

In addition, option 1) probably also want to use multiple threads, to make it scale across CPU cores. Option 2) typically doesn't need that, because each process can be scheduled on a different core.