The gist of it is to collect metrics on everything important that players do. Skagg's refers to this as "developing a metric's mindset." In order to answer questions, you should assume you don't know the answer, and instead test everything. If you want to know whether to color a link red or or green or pink, you should test it and see which gets the highest click-through. (In the example he gave, they found that pink links had over twice the click-through of red-links for a promotion that was running at the top of Farmville)
Zynga definitely follows their own advice: apparently they pull in about 2 terabytes of data on user behavior daily (which is actually optimized down from the 4 terabytes they were pulling before.)
In traditional retail games you can't get much information on what the player's are doing. So, there are a lot of questions you'd like to ask, but you can't in the traditional game world. For instance:
1. How many players make it past the installer?
2. How many players continue playing after the tutorial?
3. How many players actually complete the game?
4. What do players do the most in the game? What do players avoid doing?
... and so on ...
In a social game, most of these questions are answerable, so why not test it?
Some other interesting tidbits from the Q&A session:
- Farmville is very popular among women 35-50 years old
- Over 50% of facebook is over 35
- One of the tests they sometimes use is a cell phone test -- ie, could you play this game while you're also chatting on a cell phone? (Hardcore gamers might balk at this, but, on the other hand, 80 million players is hard to argue with)
- Zynga's focus really seems to be not on making the games they like, but more very carefully understanding the market's that they're entering into and catering something exactly for them. This is in very stark contrast to say, the indie mindset, of making something without "dumbing" it down.
- If you want a successful social game, you should really be aiming for about a 30% retention rate of players
