No, the capacity is 20x larger than you wrote

Oops, I've thought it was 15 servers is for 10'000 players. 100'000 simultaneous players on 15 (two-socket?) servers for a sim - you guys should be geniuses ;-), PLUS it should be very simple simulation, and/or lots of players who're-static-at-the-moment (which both imply significant reductions in server-to-server traffic). Still, 30-sim-servers-for-10000-players are not too uncommon.

Unless you mean that the 30 servers are simulating 10,000 players *each*

No, I don't.

Nobody in their right mind would send only a single update in a packet anywhere

Yep, I should have written "overhead-per-packet". Mea culpa. Let's also note that this remark doesn't change my 50-byte-per-update calculation. Why I've brought it up - because in similar argument elsewhere I was once told that I should add header overhead (which is crazy, and this is my point).

beyond what actual facts show

Could you re-iterate what exactly those actual facts are (beyond the fact that you've done it differently, and it works that way, which I have no doubts about, but it doesn't prove that "your" way is the only one possible, and beyond those I've already addressed - unless you can point out the mistake)? As I see it, I've addressed all of your specific concerns with specific numbers (and each of the numbers has like 100x reserve to it before it will start to cause real trouble, so 2x or even 5x difference in my guesstimates won't change much). Moreover, the "group" thing I've described above makes two deployment systems ("yours" and "mine") extremely close to each other. If you see a mistake in my calculations - please bring it forward, but I've went through them several times, and don't see any issues, so for me they seem to stand.

What's the actual benefit? What's in it for you?

The benefit is exactly as advertised - to make the book better. Despite the popular opinion ;-) , I'm open to learning things, and speaking to smart and experienced people (even when they're aggressive) often reveals some aspects which I don't know about (one cannot know everything) or didn't think about (everybody has a right to screw-up once in a while). You see, when it comes to Front-End Servers and Client-Side Load Balancing, I've done it myself for a very big non-sim game, and also discussed it with sim-MMO guys who were positive about it (up to the point of "we'll probably do it for our next game" - they did get their share of failures because of faulty server-side failure handling). On the other hand, as you keep insisting that it won't work - there should be some applicability limit, which is important to know. That's what I'm trying to find out. For your game, with 100K players on 15 (sim?) servers - it indeed might not be the best thing (though even there it depends on the question "at any given moment, what is the percentage of your players either standing still or moving strictly linearly"?), but how large is the class of such games?

BTW, while we're at it - could you tell what exactly is the mechanism you guys are using to detect-server-failure-and-swap IPs (which scripts, where they're running, and so on)? What I've seen up to now, didn't work reliably enough, but maybe you know some specifics which I don't?

could you tell what exactly is the mechanism you guys are using to detect-server-failure-and-swap IPs (which scripts, where they're running, and so on)? What I've seen up to now, didn't work reliably enough, but maybe you know some specifics which I don't?

First: The 15-server cluster is for "view" only (and chat.) The "sim" part that generates view events is sprawling, heterogenous, and not efficient -- and we don't even do physics :-) But, that means that we can use stateless load balancing! My main point was that 30 servers for "view" for 10,000 players is way too much overhead, and with the right architecture, you could get by with much less.
It's actually structured as a three- or four-tier system depending on whether you connect with TCP or websockets (good) or poll through HTTP (bad, but supported.) The first tier is fully random; it just proxies connections to the next tier based on your user id and a hash table; that tier then bounces different state updates based on topic to the back-end tier. But, as I said -- this won't work for physical simulation. In the previous system, we had the triangle of client -> sim -> view -> client, and had to make each of the legs for the client be a separate bidirectional NAT session. (The fourth tier for HTTP requests maps from session ID to session, effectively keeping a virtually-open connection to the cluster over long-polling. Ugh.)

Second: Because we load balance as described above in the described system, and don't do physical simulation, we use off-the-shelf load balancers. We used to use some Brocade hardware, that wasn't really all that reliable, and we now use a redundant pair of HAProxies and a pair of Juniper routers up front. Failure/replacement is done with standby hardware and "virtual IPs" that are taken over by whatever role needs it. (We're also looking at LVS, which is showing some promise!)

Third: In the previous system, where we did have the physical simulation, we had a "distributor" service that checked the ownership of physical locations, and mapped users to IP/Port pairs through a quad tree. (Actually, a cube-mapped quad tree, because we had a spherical planet the size of Earth ... and actually at one point ran Earth in it for a government project!) When a server died, we updated the map to point at a new IP/Port pair for some stand-by or lower-utilized server, which would take over. There was a service to make this update in our management console, and the newly elected servers would "discover" their new responsibilities and slurp in the necessary data. Shuffling all the physical state and spinning it up as new simulation took a while, so users would see perhaps a 30 second outage. Luckily, didn't happen that often :-)

So: Having built systems with, and without, physical simulation, I really feel strongly that random load balancing doesn't work well for the physical case. If nothing else, then the cache thrashing you'll get when you need to do interest management on the entire world on each view server is going to cost you a multiple compared to what kind of throughput you could get per server. Fit the update data in L2/L3 -- instant win!
Similarly, sending the stream of all PCs and NPCs and other game events to 30 separate view servers from each sim server, and receiving those same streams on the view side, is enough to not quite saturate, but at least tax a gigabit fabric. 10 gigabit would make it less of a problem, but that still adds about $1000 per port at current pricing.

To re-cap: The costs of random load balancing when using physical simulation are:

1) Network fabric -- if you like doing network block devices, or live deployments, or not living with 90% link utilization all the time, random costs you $1000 per port extra (because 10G networking.) If you're a big boy and run management on a separate physical subnet, this may be doubled. (We don't -- YOLO!)

2) Server throughput -- if you can fit the part of world state you care about and filter against in cache, and stream-generate update packets for each connected player, you will get some multiple of hardware efficiency out of that, saving 2x, 3x, or even 10x on number of servers you need for viewing. Which, in turn, would help with 1 as well... There's also some CPU cost to send the same network stream from the sim server to 30x the view servers.

3) The scalability of the system is not linear! As you add view servers, and/or sim servers, the overhead does in fact scale as N-squared. If you plan to build a bigger world with more players and more action, suddenly what you might have been able to shoehorn into your current infrastructure will need to scale 4x in cost to serve 2x increase capacity.

That's why I believe the industry practice is game-world-geographic sharding for both simulation and view services. It's not because we're old fogeys who can't believe that anything else might be better -- it's because anything else is actually worse!

Wow, thanks a lot for the really long post. While am I still not completely agree with you, at least I know where you're coming from (I hope).

Now my own recap:

1. I don't argue with your points in general. However:

2. I'm arguing that they will come into play later rather than sooner (in other words, in most cases they won't become a problem at least until we have 5-10 view servers)

3. When it becomes a problem, we can play a kind "hybrid" game (in-between "yours" and "mine", the one which I've mentioned before under "groups")

To be very specific, let's see an example where there are 30 sim servers and 15 view servers. As I understand (correct me if I'm wrong), in "your" architecture you would have (very roughly) each 2 sim servers connected to one view server. In other words, you'd have your system built from 15 "groups" consisting of 2+1 servers each.

Now let's consider a "hybrid" system. Let's assume that one of those problems doesn't allow to have each-sim-server-speaking-to-each-view-server configuration (doesn't matter whether it was cache thrashing, network, whatever). Ok, we're organizing it in the following manner. We're splitting the whole thing into 3 "groups", each having 10 sim+(5+1) view servers (one view server being group spare), and making a "hybrid" two-level connection selection: first, we're selecting a "group" (by location of the player in virtual world, just like you do), and then we're selecting randomly view server within a "group". In a sense, it is pretty much the same as in "your" architecture, but with "group size" being larger. If 3 groups is not enough - we can have 5 groups, 8 groups, or (in the very extreme case) even 15 groups. Whatever the number of group is, this approach eliminates all those nasty effects (there is no longer a need to do "interest management" over the whole world, but only over fixed-number-of-sim-servers, there are no n^2 effects anymore, it scales perfectly linearly by adding more groups of the same size, and so on) - and it still has all the benefits of random balancing.

I don't want to go into a lengthy discussion of pros and cons of both systems here (they're not directly comparable, especially if we take into account things such as DDoS attacks - I've been under multi-day DDoS once, and should tell it is quite an unpleasant experience). What I want to say that these two systems (one "yours" out of 15*(2+1) servers, and another "hybrid" out of 3*(10+5+1) - or 5*(6+3+1) servers or even 15*(2+1+1) servers), are actually quite similar, and therefore both will work. If you have objections - let's discuss (maybe you have some arguments I didn't understand yet), but at the moment I don't see why at least one of those "hybrid" configurations won't work (except, maybe, for some really really fringe cases). However, I suggest to leave aside a question which of them is better (a.k.a. "why in the hell you'd want to do THIS?" ;-) ) - as soon as both are viable, the choice is a judgement call, and arguments about pros and cons will lead us to a Really Lengthy discussion taking way too much time (and depending on personal experiences a lot).

Now to more minor things.

Failure/replacement is done with standby hardware and "virtual IPs" that are taken over by whatever role needs it.

Are these "Virtual IPs" provided by Junipers (and they have an API which allows you to remap them) - or something else?

My main point was that 30 servers for "view" for 10,000 players is way too much overhead, and with the right architecture, you could get by with much less.

I think we're speaking about the numbers which are too incomparable to each other (game specifics cause very different types of traffic even for sim games). What I can say that for view servers, as a rule of thumb, you can serve hundreds of thousands packets per second (whether TCP or UDP). I've done it myself and seen done it by others too. And then, with this (still very rough) number at hand and knowing the number of updates per player per second for specific game - we can speak of number of players per view server. What about this kind of approach?

And one more question - when simulating, how did you paralellize your world onto multiple cores within the same server? Was it about classical zones (with/without overlaps and/or dynamic stuff) or it was something more exotic?

1. I don't argue with your points in general. However:
2. I'm arguing that they will come into play later rather than sooner (in other words, in most cases they won't become a problem at least until we have 5-10 view servers)
3. When it becomes a problem, we can play a kind "hybrid" game (in-between "yours" and "mine", the one which I've mentioned before under "groups")

And I argue it's just as cheap to build it with per-sim view services from the beginning. In fact, most games build the view straight into the sim, because that's the simplest. Breaking out view to a single repeater is much simpler than having to stream all the view data to all the view servers (and managing which of them are up, and updating the list of the ones you have to stream to, etc.) I see very little simplification in trying to do random load balancing, and I see a lot of future cost. Basically, the one thing you could gain is simplifying the lookup "I'm on sim X, where should I get my view data from?" but you have to solve the very similar "I'm on sim X, where do I send my user input to?" and you can just piggy-back the second answer on top of that.

So, to let me understand where you're going with this: What do you believe that actual gain is in trying to do web-style random-assignment load balancing for view data?

Are these "Virtual IPs" provided by Junipers (and they have an API which allows you to remap them) - or something else?

It's a script that does ifconfig, ARP flush, done! Why complicate things? :-)

I've been under multi-day DDoS once

That's a sad fact of the world. Everyone who gains any kind of profile will get hit. We were under siege for a month (!) before they gave up once. From that, we learned that a DDoS mitigation vendor that lets you talk straight to their engineers is much more valuable than any other vendor :-) We also learned that they are pretty good at mitigating standard protocols (like HTTP) and certain common attacks (DNS reflection, SYN floods, etc) but that mitigating custom protocols is not in their comfort zone (which means all games unless you tunnel through HTTP.)

Then again, 10G unmetered transit is just a few thousand dollars per month these days, so we just got a few of those into the colo and have just swallowed most smaller probes from "rage booter" and "I have Google Fiber and I'm pissed" type attacks since then :-)

when simulating, how did you paralellize your world onto multiple cores within the same server? Was it about classical zones (with/without overlaps and/or dynamic stuff) or it was something more exotic?

The application was heavily single-threaded. We hosted multiple world slices on the same physical host. We had a design for actually parallellizing a single world zone (structuring physics/sim state to always read time N-1 for the surroundings when calculating time N,) but never had to implement it. (That company got bought out by an East Coast defense contractor, and I like the West Coast, so I didn't stay around after that.)

What do you believe that actual gain is in trying to do web-style random-assignment load balancing for view data?

First, let's be very clear here. What I'm arguing for, is NOT a classical-web-style-load-balancer-box-on-the-server-side (and right, having it wouldn't have any of the benefits I'm after). Random balancing is done by client, with no hardware on the server side for this purpose, this is important.

Now, as I've said, I don't want to go into lengthy discussions on gains, as it can easily can take forever and even longer. However, very briefly, two practical benefits of purely client-side load balancing I've observed myself are the following.

1. better overall reliability (this is due, in particular, to "false positive" problem I will mention below). And as a personal experience, a story in this regard: once upon a time, our system-with-client-side-balancing went through the technical audit (it was BMO's top technical guy, IIRC). And when we gave the numbers of the downtime, the auditing guy said "it is 10x lower than the industry average, it's too good to be true!". We gave him the logs, and he eventually was convinced, but the moral of the story is different: whatever we've done, reliability was Really Good compared to the rest of the gaming industry. And with all the problems which plaque server-side detection and systems marketed as "no-SPOF" which fail in practice badly (usually admins saying "it was due to misconfigured script", as if it helps), I tend to attribute it at least in large part to client-side balancing (which is inherently bulletproof, as soon as you get those 5 lines of code doing it, right). Yes, your experiences may be different, but mine are like this.

2. Significantly better resilience to DDoS attacks. And I'm not speaking about one disgruntled customer, but about massive things which bring your whole ISP down (or are bad enough so that to avoid affecting other customers, ISP imposes inbound packet limit on your incoming traffic, which is the same thing from your perspective). Then, client-side balancing over two different ISPs (opposed to any kind of server-side balancing) provides a Big Fat Benefit: to succeed, the attacker needs to bring both of your ISPs down. And I've never seen (or heard) of an attacker who managed to do it (while single ISPs do restrict per-customer traffic due to DDoS on regular basis). Speculations on why it happens: it seems that most of the time attackers just don't realize what is going on, and if they do - it is usually too late to redirect the attack (as they need to keep away from their control centers for security reasons, so the attack goes pretty much unsupervised). Also, bringing down one ISP is a challenge per se, and splitting attacker forces between the two (without real knowledge about ISP's relative capacities) is not exactly trivial. More on making DDoS attacker to play whack-a-mole, below.

Of course, your experience and priorities may be very different, but for me these two advantages of pure client-side balancing are very important.

It's a script that does ifconfig, ARP flush, done! Why complicate things? :-)

Damn. Honestly, I was hoping you'll tell something better than this one (which BTW would stronger the case for server-side failure handling). There is an inherent problem with this approach (and I've seen and heard about it failing quite a lot). The problem goes as follows: if you're swapping IPs like this, then any kind of "false positive" in your server-failure-detection script (and such "false positives" are pretty much inevitable, as any failure detection is based on some kind of heuristics, and server failures are not guaranteed to be all-or-nothing), will lead to the old server keeping IP, and new server receiving the same IP. Which usually means that both servers will be pretty much dead (some traffic will pass intermittently, but it won't be playable). Been there, seen that. And heard about that A DAMN LOT. Once again, YMMV, but from what I've seen, it adds to real-world reliability issues. EDIT: A thought has just crossed my mind (no idea why nobody-I-know does it) - if in case of failed-server we physically (or at least at iLO/DRAC/IMM level) turn it off (it has supposedly failed anyway) - it would solve the problem (or at least mitigate it greatly). Did you guys do something along these lines?

From that, we learned that a DDoS mitigation vendor that lets you talk straight to their engineers is much more valuable than any other vendor :-)

Well, from my experience it is much better to have an ISP which lets you talk straight to their engineers (who know what they're doing and what DDoS is about). You won't believe how much of the average attack can be filtered out simply at ISP ingress level (and it will save the ISP from overloading too). One very first thing to do (that is, if you have TCP as a backup connection option, and clients auto-try it in case of UDP problems), is filtering off all incoming UDP traffic aimed at your servers at ISP ingress filter; as UDP tends to constitute 90+% of all the attack volume - bingo! 90+% of the attack traffic is gone in a jiffy (and you're still alive and kicking, while with degraded quality of service, but for quite a few RPGs it can be still playable over TCP).

Overall, when dealing with a (serious) DDoS attacker there is nothing better (or more satisfying ), than making him play whack-a-mole with you . He's hitting one of your IPs - your clients are automagically reconnecting to the others within the group, he manages to hit the whole ISP - your clients are reconnecting to another one, he's hitting UDP - you're off to TCP, and so on. And as controlling botnets in real-time is not what is normally done (even less so for crowd-attacks), chances that they give up before they figure out an optimal attack, are very high.

The application was heavily single-threaded.

Thanks! BTW, under "heavily single-threaded" you mean something along the lines of shared-nothing actors (Erlang- or Node.js-style) - or something different?

We had a design for actually parallellizing a single world zone (structuring physics/sim state to always read time N-1 for the surroundings when calculating time N,) but never had to implement it.

Thanks for mentioning it. It sounds somewhat similar to what Halo/Destiny does on the client side. Well, the problem is somewhat similar, so solutions should be somewhat similar too.

Random balancing is done by client, with no hardware on the server side for this purpose, this is important.

That has two other problems:

1) How do you get the IPs to choose from to the clients? What happens when your IPs get relocated?
2) If you have N view servers, you need N public IP addresses. This is actually becoming a real problem for the IPv4 crowd. If you need 30 view servers for a 10,000 CCU world shard, and you run 100 world shards, that's a very large chunk of IPv4 space to get ahold of and manage.

And I'm not speaking about one disgruntled customer, but about massive things which bring your whole ISP down

Yes, that's why you have multiple ISPs. And, when that's not enough, you BGP swing your IP subnets to a mitigation vendor with a terabit of ingress, they scrub it, and forward over a GRE tunnel or something to your own network.
Trying to play whack-a-mole by filtering at the ISP was something we did in the early days, but the trolls have more patience than we do -- we have a business to run! I really don't recommend it -- it wasted a bunch of time and service quality.

Even single disgruntled customers have bitcoins, and know about botnets-for-hire services.

BTW, under "heavily single-threaded" you mean something along the lines of shared-nothing actors (Erlang- or Node.js-style) - or something different?

No, it was all custom C++, and it used a number of static variables and other non-locked data structures that were not feasible to multi-thread in-process. (The simulation code started out before multi-core was a thing!)

It's a script that does ifconfig, ARP flush, done! Why complicate things? :-)

Also, note that this is on the inside network, behind the Junpers that run reverse NAT. The public IP doesn't change at all.

Which brings up the point: Even if you have client-side load balancing, picking from some set of IPv4 addresses that you somehow robustly get to the clients (through a patcher using separate CDN or whatever,) you will have some point of entry into your network, which will likely be some redundant pair of routers that do edge routing, connect to your ISPs/uplinks/transit, etc. Those boxes can do load balancing (like F5 and such is known for) or they can just do routing (like we do with our Junipers) but I don't really see how using hardware load balancing would introduce a 10x robustness penalty compared to just doing routing. If someone screws up the router, they screw up the router, and it affects all traffic and gets rolled back right quick! :-)

1) How do you get the IPs to choose from to the clients? What happens when your IPs get relocated?
2) If you have N view servers, you need N public IP addresses. This is actually becoming a real problem for the IPv4 crowd. If you need 30 view servers for a 10,000 CCU world shard, and you run 100 world shards, that's a very large chunk of IPv4 space to get ahold of and manage.

From what I've seen, neither has ever been a problem in practice. As for the first one - even client updates are coming more often than IP changes/additions (I can remember only one ISP-induced IP change in 15+ years over multiple datacenters, and it was a Big Event with 2 months advance notice etc. etc.), and also there is always an option to distribute IP lists via DNS (just without round-robin, which causes severe imbalance due to asymmetries in caching). And with all the continuous delivery stuff coming in (which I don't like, but which is becoming a Business Requirement) - client updates are going to be even more frequent. As for the second one - last time I've heard, the long-standing ARIN/RIPE rule "each box has a right to at least one public IP" was still standing (5 years ago, you could easily have 10 IPs per box plus one per SSL certificate, now it is more like 2IPs-per-box and nothing-for-SSL/TLS); heck, they're still giving an IP per small cloud instance (which they run dozens per box). And it is very difficult to impose limits below 1-IP-per-box (it will amount to prohibiting buying one single box) - I don't see it happening, and massive migration to IPv6 looks much more likely than this. BTW, with all the buzz about exhausted IP space, Linode still provides additional IPs at $1/month (yes, that's ONE DOLLAR PER EXTRA IP PER MONTH) :-) (ok, before massive going into IPv6, the price will probably get up to $10, but that's about it).

I really don't recommend it -- it wasted a bunch of time and service quality.

Well, from my experience it worked perfectly for a Really-Large ($1e9 revenue/year) and Really-Attack-Prone (at least because of size and amount of money involved) online-game business over a dozen of years. All the single attackers were non-issues at all (no reaction was necessary, rebalancing occurred automatically just because of the nature of client-side balancing, service degradations were observed only at instrumental level), and the only cases where intervention was required, were those organized attacks when the whole ISP is at risk (and these are fortunately very rare). The worst thing which I know about, was an extortion attack which has affected the whole ISP (which was investigated by Scotland Yard and where the attackers were reportedly jailed 2 or 3 years later). As for botnets-for-bitcoins - sure, they can buy it, but as soon as the attacker starts an attack and sees that you're not affected - they almost-universally call the attack off very quickly. Bringing down the whole ISP is not something you can do for cheap, and spending money without seeing any effect of the attack is outright frustrating.

you will have some point of entry into your network, which will likely be some redundant pair of routers that do edge routing, connect to your ISPs/uplinks/transit, etc

Ah, now I start to see what exactly you keep in mind. Yes, it would work, but I should say such setups are highly unusual at least for start-ups. When somebody is starting a new business, most of the time they will be renting servers (colos are too expensive/too much headache/too difficult to expand/...) - or even renting cloud instances (which is usually preferred if not for latencies). When you rent servers (rather than effectively running your own ISP with your own routers, your own BGP, and your own AS(!)), all the routers belong to your ISP, and you can't really affect how they're configured (it is almost-universally considered an ISP's job to provide service at the point of Ethernet-port-coming-out-of-their-router, how they're doing it - is almost-universally beyond the scope of your SLA). Making them configure routers for you is usually not an option (unless you're that $1e9 business), making them use routers-which-you-prefer is hopeless. Having your own routers behind their routers (which you seem to imply) is possible, but is very unusual (I've seen games with their own AS, but it wasn't until the whole thing was already Very Big and Very Profitable, and it takes quite significant time to get there), not to mention quite expensive (some cheaper ISPs won't even let you get your own hardware into their racks, and those which will - will charge you a lot just for doing it). Rented servers are cheap as dirt, but any deviations from whatever-your-ISP-has-in-stock (and usually routers are not on the menu) tend to increase pricing really badly.

If someone screws up the router, they screw up the router, and it affects all traffic and gets rolled back right quick! :-)

The more configuration is needed - the more chances to screw up they have :-(. Also misconfigurations at that level are very often unnoticed until the fault event occurs (and even then, the problem can be with only some of the servers, sitting deep within). Human errors are one of the Largest Sources of problems, so keeping configuration at the absolute minimum does help. As for the 10x better reliability - well, I can't tell why exactly it was 10x better, but I tend to attribute a significant portion of it to less-opportunities-for-human-error.

NOW, let's get to a really interesting stuff :-).

you BGP swing your IP subnets to a mitigation vendor with a terabit of ingress, they scrub it, and forward over a GRE tunnel or something to your own network.

Ah, so you're speaking about this kind of DDoS vendor (and not about in-house DDoS systems sitting within an ISP next to your servers, which I am not fond of, to put it mildly). TBH, I've never had hands-one experience with one of these, So, if you have time (and can) - please tell me everything you know about these guys ;-). In partucular: is there a good description on "how it really works" (beyond marketing B/S a-la "we will do everything for you by some kind of magic")? What kind of cooperation is needed from ISP (probably they need to allow you running your own AS within them, and maybe something else), and how willing ISPs are to do it? What is the smallest IP range which can be reallocated via BGP (yes, my knowledge of BGP is very rudimentary)? How long does it take for BGP to converge for such unusual reallocations (is it usual 1.5-2 minutes, or there are some differences)? How much such DDoS services cost? Which vendor you have experience with (if it's not a secret)? Were they good? How much additional latency do they introduce when you're under attack? What are the typical detection/reaction times? What about collateral damages (false positive rates, in terms of legitimate-customers-thrown-off both while not under attack, and while under attack)? And anything else you can think of :-)

Yes, it would work, but I should say such setups are highly unusual at least for start-ups.

So, in one breath, you talk about a $1e9 enterprise, and in the next, you're talking about a start-up. I agree that those have different requirements. But if you want to build a WoW-level MMO -- or even just a Tera-level MMO, or just an Everquest analog -- start-up approaches won't get you to the goal. YOu *could* rent everything you needed from Amazon ECC, because they expose highly advanced DNS, load balancing, etc, but then you end up paying a significant multiple on top of what it costs to run it yourself.

in-house DDoS systems sitting within an ISP next to your servers

Those don't really work for anything big, I agree. We don't even need a specific appliance to mitigate smaller DDoSes; as long as the ingress is not saturated, we can mitigate in our own infrastructure. Something as basic as SYN cookies and port filters helps tremendously for those cases.

please tell me everything you know about these guys ;-)

Unfortunately, they come with enterprise contracts that may prevent me from talking too much about them, but I can talk in general.

The vendors I know from some level of detail are Verisign, Neustar, and Akamai.
Their architectures are the same at the high level -- lots of ingress all over the world; data center with stand-by network filtering hardware; GRE tunneling to your own premises. Return traffic typically does not go back through them.
Some of them use custom kernels/software on commodity hardware; others use dedicated scrubbing appliances. Ask their sales and they'll tell you.
BGP swings start showing traffic after 10-15 seconds, and is largely complete after 30-60 seconds. It's certainly a lot faster than DNS-based solutions!

Yes, we run our own AS. Neither of our ISPs batted an eye at that. (One is small and high quality; the other is large and cheap.)
The cost for DDoS mitigation is some amount of yearly costs (thousands per month) and that includes some amount of mitigation events per year (an event might be defined differently per vendor.) Additional events cost additional money, on the order of a some thousands per.

We have not measured additional latency. I'd expect dozens but not hundreds of milliseconds.
Detection and reaction is all on us -- the vendor is a passive stand-by until we activate it. When we activate, they are generally always ready as soon as we call. So, detection time depends on our monitoring infrastructure. We take and store all measurements every 10 seconds, but need longer than that to verify it's actually a DDoS and not some other kind of "blip."

Vendors typically have canned filters for "typical web sites" but a game is different, and you need to work with them to figure out what they need to change. They are happy to make changes to your scrubbing filters, but you should make not of all the changes yourself and request them as part of the next mitigation request up front, to save time.

The cost for DDoS mitigation is some amount of yearly costs (thousands per month) and that includes some amount of mitigation events per year (an event might be defined differently per vendor.) Additional events cost additional money, on the order of a some thousands per.

How many "events" requiring actual mitigation (rather than just absorbing it) would you estimate a widely-known MMO like WoW, Destiny, or GW2 would encounter a year?

How many "events" requiring actual mitigation (rather than just absorbing it) would you estimate a widely-known MMO like WoW, Destiny, or GW2 would encounter a year?

I have no idea! I think it varies wildly based on the exact kind of user base you attract, what your business model is, and how big you are.

Curiously, the bigger you are, the less "nuisance" attacks you see, because your ingress is big enough that the small DDoS-es don't actually affect you.

I'd say we ( https://www.imvu.com/next ) get about 1-5 events per year that require mitigation, and we additionally run fire drills a few times a year where we mitigate without an attack just to make sure it still works.

Also, the cost of the contract depends on your estimated size. A contract that covers you for a Tbps attack is going to be a lot more expensive than one that covers you for 30 Gbps :-) (Our contract is somewhere between those extremes.)