Hi all,

I have two questions about the pain-point for game developers in the ongoing operations of social games.

Full Disclosure: I'm in charge of business development for English-speaking clients at a Japanese cloud computing & development firm that specializes in this as one of our service (operating social games for developers & producers).

Questions:
1. How much of a pain (and time-sink) is it for you to monitor your servers & do daily operating of a social game?

2. How interested would you be in outsourcing this work to another firm, so you can focus resources on the game content/design?

Background:
In the rapidly-growing Japanese social game development scene, we've seen a big need for outsourcing the daily operations of games. We've seen this for large AAA companies as well as for small development studios with limited resources. Social games are unique in the frequency that the backend is updated to optimize user conversions, as well as the amount of "events" and feature additions that are pushed to a game to keep users engagement and retention up.

I imagine the same desire to outsource this "grunt work" exists in the non-Japanese game development market, but I'm curious to hear all of your opinions as I know things can vary from culture to culture.

My Definition of "outsourcing operations":

When I write about outsourcing operations, I'm referring to the following:

• 24/7/365 monitoring of servers/systems
  • adding/removing servers to handle spikes in traffic
  • emergency repair if anything malfunctions or goes down
• prepping & deploying new creative content to the game (new cards, characters, etc.)
• tweaking game elements to optimize user interaction & conversion rates

Thoughts? Is this a service you would pay for to be able to focus more on the actual game development?

Thanks!

Todd Spitz | Fixer Inc.
toddspitz {a}{t} fixer.co.jp

Some would argue that core competencies should never be outsourced. From this perspective, the actual running of the game (sys admin, updates, SVN, etc) is pretty much THE core competency you need, considering you can often license a social/browser game for your specific geography.

Others might have a different take. Make no mistake, no one LIKES the sys admin tasks, there's always some spill over into development, marketing, operations. And it's almost never good news either.

Unless your game is really large scale (as in spread on at least 150 threads & a couple TB + of ram) then i see no use for this really, i wouldn't outsource the server work, just outsource the servers as rent and get a good service to begin with (private cloud type), this are really cheap now and:

1) adding/removing servers to handle spikes in traffic => This can be handled in a few clicks now from someone with close to 0 technical knowledge

2) emergency repair if anything malfunctions or goes down =>I'd rather have the big datacenter hosting my private clouds do this online without me even noticing

3) prepping & deploying new creative content to the game (new cards, characters, etc.) => this is unrelated to technical operations, i would certainly not offload at the same time marketing, content creation and technical operations to the same company in today's highly specialized world!

4) tweaking game elements to optimize user interaction & conversion rates => same as 3.

• adding/removing servers to handle spikes in traffic
• emergency repair if anything malfunctions or goes down

Those are normally done, for the hardware, by the datacenter (without explicitly outsourcing anything, apart from the fact that you rent the server there).

Everything else, i.e. setting up your server software on the newly installed machines, maintaining the database, etc. I wouldn't even dream of outsourcing. It is already a considerable responsability (and... risk!) to have this kind of power assigned to someone who is not a company owner in-house. Someone who is a shareholder has a very firm interest in keeping things up, but for everyone else that isn't necessarily the case.

Just imagine what happens, for example, if you have to fire your database admin, and he deletes your user accounts on the way out of the door (and of course there are no backups to be found). Or, imagine he takes a copy of the password database with him. You can sue him, sure, but to what avail. Where there is nothing, the king loses his rights.

Imagine he sells your user activity logs to the Yakuza to pay his gambling bills. You'll find out when it all blows up and the feds knock at your door. Yes, someone else did it, but it's your responsibility. Yes, you're still going to jail for it in the worst case.

And now imagine this is outsourced to another company. You might not even know for sure who exactly deals with your data. You might not know how many people know the root password. Maybe it's written on a yellow sticky that some guy pins to his monitor. That company might outsource the work on their own part, possibly without you knowing, and possibly in a location that you wouldn't like. They might not do some things properly to cut down cost (say, backups) which you only learn about the moment you need to rely on them.

They might go bankrupt tomorrow, and the liquidator might seize all that's in their building, including your stuff. Of course you have evidence that it's your stuff, don't you. So it will only take 8-10 months at court until you get back access to what's rightfully yours, no sweat. Hey, who needs to make profit anyway.

Another vote for keeping it in-house and renting a data center.

There are systems like EC2 (and many others) that provide fault-tolerant global redundancy and automatic scaling.

You can get a full month of EC2 for $43. That's an online virtual machine with 1.7GB memory and 160GB of storage space. If your needs grow --- either slowly or in a short burst --- the system will automatically scale up and automatically load balance for about two cents per hour.

For a popular social game you may end up paying a few hundred dollars per month for the data center, but that is still cheaper than outsourcing your core development process.

Hi All,

thanks for the input and insights. I now realize that I was a little unclear as to what we do and who our target market has been, but all of your insights are still helpful in understanding the mindset outside of Japan a little more.

To clarify on a few points:

?We are full proponents of using the cloud. We work almost exclusively in the public cloud (specifically Windows Azure) because of it's relative stability, geolocated server redundancies, scalable resources for traffic bursts, and its cost-effectiveness for our clients.

?Our approach in Japan has been two-fold:

1. Server-side development — we are specialists in the cloud & proper server development. While a lot of game companies we've worked with are good at game design and the front-end programming of games, we've found they often lack the expertise to optimize their systems for low latency, proper scalability, etc. This naturally includes games with global distribution, as latency becomes a bigger issue.

2. Ongoing operations — public cloud or private cloud, things still malfunction & services become unavailable, so we step in as soon as anything needs resetting/repairing/etc. As a close partner of Microsoft here in Japan, we have exceptional insight into any issues with their datacenter operations as it happens.
Regarding content deployment, sys admin, updates, SVN, etc., we do this for clients as it is a mission-critical task that must be done, but no one enjoys doing. To be clear, we're not usually the ones developing creative content or updates—we receive the work, prep it for release, test, deploy, and monitor.

I understand samoth's concerns about access credentials, and this is something I will have to think about more for the non-Japanese market. Because we are operating major social games for Square Enix, GREE, Mixi, etc. with 100K—3M+ users, the stakes are way too high for us to allow anything like data theft, unauthorized database tampering, etc. to happen (and we take careful measures to prevent it), but this is definitely a reasonable concern and something that may be harder to convince people outside of Japan about even if we have a strong record with top firms in this country.

Some of this comes down to a target market question, as I understand our services are not for everyone. We are definitely most helpful for companies that excel in game development and would rather spend company resources on the actual game development and adding new features/creative content to a game vs. worrying about the server environment.

To alter the question a little: of your current backend server/cloud operations, which components (besides the physical datacenter) would you consider outsourcing if you found a company you genuinely trusted in their abilities & safety?

To alter the question a little: of your current backend server/cloud operations, which components (besides the physical datacenter) would you consider outsourcing if you found a company you genuinely trusted in their abilities & safety?

For all the technology companies I have been at in nearly two decades: NONE OF THEM.

That technology is the core to the business. The technology business creates the front end that faces the user, and it creates the back end that stores the data. Those two things are the core of the business.

When you outsource your core business it is no longer your core business.

When a game development company starts outsourcing their core business at that same moment they cease to be a game development company; they become a publisher rather than a developer. Instead of developing their own work they are a publisher of other people's work.