Are there any must-have DoS / DDoS mitigation strategies one should always build into a server?

Regardless of TCP or UDP, it feels like there is very little one could do against a DDoS which tries to saturate the network.

Even if the attacker can't do that, simply sending login packets with spoofed IP addresses/ports could be a problem. For something like SSL, it's possible to hit the CPU hard by initiating a handshake. CPU exhaustion attacks can be mitigated by puzzle challenge-style logins, but it should be fairly easy to block login by making sure that the all login "slots" are in use.

(I know some websites that use CloudFlare, but that's for serving http/https content and so isn't an option)

Any opinions of what a reasonable line of defence is?

With the mentioning of cloudflare and in AllEightUps post number 4 he uses a ssl connection for initial id/salt I saw recently that the new beta firefall uses simple https requests to aws servers for login. You could do similar use cloudflare to prevent alot of attacks on the initial login atleast. You would still have to worry about the bandwidth issues after logged in though. Depending on what kind of game it is if its not real time you could possibly get away with all http requests using the cloudflare or other technology thats already in place.

The ability to detect remote IPs that issue an unnormally high rate of requests, or an unnormal distribution of request types, is useful.

Also, hardening your system and setting in place upstream filtering is useful. If your ISP can throw away UDP traffic destined to you on ports that you don't care about, then your link doesn't need to worry about this. (The ability to do this varies based on the ISP and your relation to them, i e how good a customer you are.)

Making sure you have SYN cookies turned on on your servers and/or load balancers is common sense. Same thing for filtering obviously bad IP fragments, etc.

Regarding CloudFlare, they are a mid-tier DDoS vendor, worrying more about CDN than DDoS, so they don't have a lot of ability to work with custom protocols like games. I think you'd get the same from other CDNs like Akamai.

There are higher-end vendors, where you pay five figures a month plus possibly five figures per event to help mitigate attacks. Those vendors may be able to put in place custom filters that you and they develop together, and can target any services. Names that come to mind include Prolexic, Neustar, and Verisign. Some of those guys even claim that they started out specifically as DDoS mitigation providers for MMO games!

Another option is to run on a public cloud, such as Amazon ECC. Yes, you pay for bandwidth, but if the DDoS is some number of gigabits for a few hours, that's not actually going to accrue all that much traffic. On the other hand, on ECC, a determined attacker will be able to generate pretty big bandwidth bills for your service... and Amazon may find that your service disrupts other customers and cut you off if it gets bad enough.

Finally, you can get high-speed connections to your ISP, with a lower commit level. For example, you might be able to find an ISP that lets you commit to half a gigabit per second, yet allows 100 Gbps interconnect, and bills by the 95th percentile. You need to be under DDoS for almost two days, full time, to actually make the DDoS hit your 95th percentile. Given that renting out botnets is a lucrative market these days, somebody has to be really pissed at you to hit you with dozens or even a hundred gigabits for more than an hour. I've only really heard of that happening to the financial services people, presumably by black op teams from their competitors or extortionists.

Good ideas all around. Some questions:

AllEightUp:

For bandwidth-limiting, you're thinking of a simple thing to prevent the client from issuing requests and getting response from the server for the request, right? Because nothing's going to prevent the packets actually hitting my server. Or are you thinking of some configuration I could do to firewalls?

For UDP I have encryption, but only after the UDP payload is spliced into sub-commands. I don't have it on the UDP protocol level. I don't know if that's a problem or not. For TCP the encryption happens after I split the stream into packets, but that feels "safer".

jeff8j:

Doing login over https has occurred to me exactly for that reason. Right now it uses a puzzle-challenge and limited evaluation pipe to prevent overload. Still, that won't protect it against bandwidth saturation. On the other hand, say I go with something like aws, then this might even make it worse. After all, killing the login server with requests is fairly harmless. However, if they fail to do so, or are able to get a large amount of login tickets, then they might start targeting the lobby and game servers - and those are the ones I would like to protect.

I've made secure login for game/lobby as cheap as possible by relying on a kerberus-style ticket to set up encryption, but they are hitting the db after the security handshake... The nice thing about the login servers are that they don't affect any sub systems if overloaded.

It's hard for me to tell if moving the login to a simple https service would help or not.

hplus0603:

What would I gain by filtering but my game port? Assuming I have a firewall closing everything else, what would they get by bombarding other ports, as opposed to simply the game port?

There are also other things you need to be sure to handle, one of the latest and greatest attacks is the simplest of all:

A pseudo slow client.

The trick is to establish TCP connections, valid ones, but send as little data as possible. Since data is trickling over the connection, the bad connection remains open, consuming a great deal of resources. It is, in effect, the same as a TCP Syn flood attack, except the connections actually complete.

There are, of course, the other things you need to handle:

Validate everything, never trust data coming from the client.

What would I gain by filtering but my game port? Assuming I have a firewall closing everything else, what would they get by bombarding other ports, as opposed to simply the game port?

A software firewall on a port can "help" to a degree, but the machine still ends up having to handle the data getting sent to it. While the data is simply discarded, it is still being sent to the machine.

Hardware firewalls are a bit better on this, since they're actually designed for throughput and packet streaming, so they can tend to discard blocked traffic at a much faster rate, but again... that's still traffic ON THE WIRE. If you're getting charged for bandwidth, you'll get charged regardless of if you accepted the traffic or not.

I've made secure login for game/lobby as cheap as possible by relying on a kerberus-style ticket to set up encryption, but they are hitting the db after the security handshake... The nice thing about the login servers are that they don't affect any sub systems if overloaded.

Who is hitting the database? The client should never touch the database. In fact, the service the client talks to should, ideally, not talk directly to the database. The database should be firewalled, locked away behind services that do all the intermediate actions. Those services should be located on servers that are locked away from the rest of your machines, if at all possible. The simple fact is, a server on the internet is a gateway. Once someone gets into your gateway, then you need to limit their lateral movement. A database server sitting on the same network as said gateway is a ripe target to get its data ripped. But a database server locked behind some service gateways and firewalls is a much harder target to access and would give you time to notice and stop said infiltration.

Washu: In regards to the port blocking, say I have the game port at 6000. What does an attacker gain by flooding 6001 that the attacker can't gain by flooding 6000? That's what I was wondering.

In regards to the db - it's the server that connects to the db of course, I was referring to the fact that a login / request by the client may cause the server to perform a db query.

In the case of a DoS attack on a server, then that server will trigger many db requests. If the queries are good and the server has a connection pool that limits the number of simultaneous queries to the db, that shouldn't be a problem. Still, I prefer to avoid enabling a player to cause a db query as far as possible.

Like you said, the db should never be accessible directly from the internet, and preferably thoroughly locked away inside the internal network.

In my case I was imagining the setup looking a bit like this:

Washu: In regards to the port blocking, say I have the game port at 6000. What does an attacker gain by flooding 6001 that the attacker can't gain by flooding 6000? That's what I was wondering.

Not much, other than increasing your traffic bill. They're much more likely to attempt to flood an open port than one that's closed. Mainly because the goal is to swamp the machine, and the easiest way to swamp a machine is to use up the kernel's free memory by clogging it up with useless garbage through open ports.

In regards to the db - it's the server that connects to the db of course, I was referring to the fact that a login / request by the client may cause the server to perform a db query.

In the case of a DoS attack on a server, then that server will trigger many db requests. If the queries are good and the server has a connection pool that limits the number of simultaneous queries to the db, that shouldn't be a problem. Still, I prefer to avoid enabling a player to cause a db query as far as possible.

Like you said, the db should never be accessible directly from the internet, and preferably thoroughly locked away inside the internal network.

In my case I was imagining the setup looking a bit like this:

Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.

There would be a DMZ for the proxy servers, and then an internal network for the game logic servers.

Lerno:

Using https wouldnt protect the game servers directly but could do woners as far as login and lobby systems. With the pro cloudflare you can set the cache to 30 seconds and they have servers all over the world so when people hi the lobby they can bring up the cached version 30 seconds isnt that much of a difference then the cloudflare servers would only touch your server once every 30 seconds. You would probably want to whitelist the cloudflare servers and block anything else through a firewall. Also the nameservers would point to cloudflare not your ip so you can keep your ip for login/lobby not known to the public. You could also have a fall back server or multiple if some get in trouble. You would also have the servers report over https to keep that login/lobby server masked. Overhead would also drop as you dont have to serve https from your server you could serve http and have cloudflare do all the heavy https work.

Note: If your game could use websockets you could block ips and cloudflares ddos would still work to prevent them from even touching your servers. This does have the downside of its not a direct connection so latency will go up a bit but I have done this and its not that bad I figure because cloudflare trys to be as close to the end user as possible so its pretty much in line on the way to the server.

In the end the only way to be slightly confident that things wont go down is to have enough servers in enough locations with big pipes that it would be very difficult to bring them down and if one goes down another handles the load. Thats how google, facebook and the like handle things and thats why a web option is easier to get going something like cloudflare has many powerful servers in place around the world with massive pipes I know I couldnt afford.

Of course there are alternatives to cloudflare some way better but none at the same level of cost $20 a month or even free for basic protection so thats why I keep going back to it.

Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.

That sounds like a lot of work unless validation/packet validation is heavyweight - and that should depend a lot on the type of game. I personally worked on poker server backends for a few years and I never ran into a need to do something like that.

That said, the same company worked on a "next gen" poker platform which distributed everything, even worse than what you describe. Scalability and reliability was in the dust for that platform.

So to me that solution sounds like quite the trade-off.

I'm also wondering what sort of attack you'd do on a game server with a single open port. Yes, it's possible to overload it, but killing the proxy by overloading it would amount to the same thing, wouldn't it? Please explain the benefits, because I don't really see them.

As to the encryption, I don't generally trust myself to get all the edge cases correct and rather use the separate SSL connection for initialization. The ssl connection is only used for login and other secure items. As a benefit, you can use it for the initial NAT traversal since you have the connection anyway. In general though, this tcp connection is only there for a brief period and won't interfere with the UDP once initiated.

That's good advice in general. I've dug deep into cryptography to design a protocol which I feel fairly confident in. Mostly because it's basically an implementation combining two well known protocols. Still, I know it's a risk.

(Even if it's broken, I don't actually use any user generated passwords. In addition, gameplay is mostly anonymized. It's not really possible to determine the e-mail of another player, and since game is partitioned into separate worlds, if you get hold of someone's credentials it's unlikely to be someone playing in the same world as you.)