This is rant pasted from my blog There is a lot of controversy when it comes to talking about Metacritic and its effects on the Game Industry. People have spoken about its negative impact before, Adam Sessler ranted about it back at

">GDC this year. Yet, it seems the number of developers, and (perhaps more importantly) the number of publishers that quote Metacritic is continually on the rise. Why is this a big deal? Who cares if anyone talks about Metacritic scores? The problem is this; Increasingly in our industry, people are preferring to talk about the Metacritic score of a game over anything else. That means things like technical accomplishments, artistic merit, public opinion, even sales of game get overlooked. Why bother doing any research? This number Metacritic shows me tells me everything! The first issue is how Metacritic compiles various reviews, rounding all scores to a 100 point system. As shown

">here, this is a flawed system. Not many people I know would say a B- equals a 67 and an A- equals a 91. Not even the school system. The issue becomes a little more dangerous when publishers begin making business decisions with these numbers. Many publishers base the success of a game by looking at its Metacritic score. Is that really what makes a game successful? What about the actual opinion of the reviewers? What about the opinion of the gamers? These things are of course important, but in the eyes of those at the top, it's money that equals success. Making games is a business, and every business is about making money...Wait, why in the world then would they look at a Metacritic score? With a little bit of research you can find that just because you have a highly rated game, doesn't mean anyone bought it. Take a look at The Orange Box for 360 and compare it to Call of Duty 4. Both are multi-platform first-person shooters that involve established franchises and received a good amount of marketing buzz. In fact, by looking at the Metacritic score, you could argue that the The Orange Box is better because of its 96 rating compared to CoD4's 94. However, when you look at the sales for the 360 version of this game, you find a very different picture. Orange Box sold around 800K, while CoD4 sold nearly 7.5 million. Judging the games purely by their metacritic scores could never predict such a difference. Another game I usually mention in this debate is Earth Defense Force 2017, and its rating of 69. Most of us would look at that score and judge it a failure. However, everyone I know who has actually played the game say it is one of the most fun games they have played on their 360, and I agree. So what about those titles that have high scores also for 360. Both are multi-platform shooters, have established franchises associated with them, and have had decent marketing exposure. Looking purely at Metacritic scores, Orange Box should in fact be slightly better with its 96 rating, while CoD has a 94. Yet, when one looks at the sales for the game, one finds that The Orange Box sold about 800k and CoD around 7.5 million, both on the 360 alone.and high sales? There are exceptions but it is true that most highly scored games also sell well, and this is where the trouble begins. The really important thing to remember here is this: Correlation does not equal causation. Just because a highly rated game does well does not mean it did well because it was highly rated. 1. A game isn't good because it has a high metacritic, it has a high metacritic because the critics liked it. 2. Your average gamer isn't going to buy a game because it has a high metacritic, they are going to buy a game because their friend said it's fun 3. A game doesn't sell well because it has a high metacritic, it sells well because it has a high amount of marketing exposure. I'm not saying that Metacritic is evil, or that it be cast to the depths of the earth. I use it too from time to time to get a quick general feel for the reception of a game. What I am saying is don't let your research end there. Never formulate an opinion of something based solely on a number. (unless of course you are a mathematician) READ some of the reviews, watch videos of the game, talk to people that played it. If you are up for it, you could even play the game yourself.

Quote: The problem is this; Increasingly in our industry, people are preferring to talk about the Metacritic score of a game over anything else.

Actually I believe the most important problem is the companies who determine whether or not to pay out bonuses to developers based on Metacritic rankings. This causes a chain of effect where everyone -- developers, publishers, and press -- are scared to death of ever being honest because they know that they're affecting people's pay checks over something as trivial as a review (which is just one persons opinion).

A developer feels pressured to act like a PR salesman to the press because they're frantically trying to get higher scores so they can get paid, while the press feel guilty about not liking games or get so fed up they become overly antagonistic, and publishers pull all sorts of dirty stunts to avoid being covered by people who don't give them good scores.

That said, Metacritic is not going to go away and if you're the sort of gamer who is actually interested in a hidden gem like EDF2017 then you probably know where to look to find out about it anyways.

So basically blah.

Quote: The issue becomes a little more dangerous when publishers begin making business decisions with these numbers. Many publishers base the success of a game by looking at its Metacritic score. Is that really what makes a game successful? What about the actual opinion of the reviewers? What about the opinion of the gamers? These things are of course important, but in the eyes of those at the top, it's money that equals success. Making games is a business, and every business is about making money...Wait, why in the world then would they look at a Metacritic score?

What makes you think that solely base their business decisions on Metacritic's scores? In fact, your link to the Sega story tells me the very opposite. They come to a conclusion based on information from several resources, and one of them happens to be Metacritic. I see nothing wrong with that. Since their success is measured in dollars, it's fairly easy for them to see whether they're using the right strategy.

Quote: Orange Box sold around 800K, while CoD4 sold nearly 7.5 million. Judging the games purely by their metacritic scores could never predict such a difference.

That's why any serious publisher would never base their decisions on a single magic eight ball but use the whole rack. I have to say though, claiming that OB is more successful than CoD, purely based on sales figures is a little off. We don't know the investment it took, and most of all, we don't what the target sales for both games were.

Quote: Another game I usually mention in this debate is Earth Defense Force 2017, and its rating of 69. Most of us would look at that score and judge it a failure. However, everyone I know who has actually played the game say it is one of the most fun games they have played on their 360, and I agree.

Um, you're claiming that publishers use a flawed and unscientific method of judging games and their (potential) success. Now you are using the personal preferences of a couple friends as an example and argument why the rating system is incorrect?

Quote: I use it too from time to time to get a quick general feel for the reception of a game. What I am saying is don't let your research end there.

Who are you addressing here, the publishers or the consumers? And where does this idea and fear come from, that opinions are solely based on this Metacritic system?

Quote: Original post by Wan
Who are you addressing here, the publishers or the consumers? And where does this idea and fear come from, that opinions are solely based on this Metacritic system?

The piece is intended as a reminder to other developers/publishers to keep looking at other factors in determining a game's success. The metacritic score is currently an important thing in judging a game. This comes from my own experience dealing with publishers and other developers, as well as talking to friends in other companies.
My feeling is that this number should be nowhere near as important as sales or your own personal enjoyment out of a game.

Talking about Metacritic in mainstream game dev is like talking about politics or religion. I've found everyone's opinion of it to be in some way different from everyone else.

So here's my opinion on it.

Metacritics purpose of being a review and score depository is a great idea. I can go to Metacritic and view a dozen different opinions of the game/movie and also get the opinions of everyday users. Which, while mostly written like dribble, are good indications of what normal,average Joe people think of it. It's good to have both types of reviews at your disposal.

Now. What aspects of the game industry [and other industries] use it for is a completely different issue and the core problem.

Having worked with a big publisher and listening in to how executives view metacrtic, they view the success or failure of a mainstream game in two aspects.

- Critical Acclaim [ reviews/scores. ]
- Selling Number. [ Profits. Which can be bolstered by consistently great reviews. ]

For Metacritic when they view their game on it they usually think one of three things.

100-80 score - "Yes! This game got over 80 points on Metacritic. It's going to sell great!"
79-60 score - "Oh noes! This game is a failure!"
59-01 score - "Oh noes! This game is a failure! Sack all the developers!"

The public isn't really the issue I think. This is a game of executives trying to impress other executives. Every publisher wants all their games to be the next big thing and sell septillions of a game. So they need to impress the executives of say... Walmart for instance who might buy lots and lots of that game. But Mr Walmart executive relies on his "How much is this game gonna sell?" advice from people who look up metacrtic to tell him, Instead of doing the real job of solid market research.

Is Metacritic becoming the lazy-mans market research tool? It already has! :D

But anyway, that's my completely unfounded and ranting opinion on the matter. Your guess is as good as mine.

And yeah developers have been denied bonus pay on something like a metacritic score. But I can go one further then that and tell you that around 6-12 months ago many developers where shitting their pants because publishers where using that development studios game Metacrtic score as a key determination point on whether to close down and/or let go of a company. And they did!

I think your comparison of the orange box to CoD4 (on consoles, no less!) is VERY misleading. You neglect to take into account that many people already had all/most of the games in the orange box, which would definitely hurt sales of orange box.

Furthermore, orange box has a much more pc-oriented audience, while CoD4 is VERY VERY console oriented. I read somewhere that CoD4 only sold 2 million copies for the PC. If you add all orange box and half life 2 sales together, you get 9.5 million (and that was almost a year ago -- numbers taken from http://www.shacknews.com/onearticle.x/56193). Furthermore, this is only RETAIL figures, and doesn't count the number of copies sold on steam.

"A game isn't good because it has a high metacritic, it has a high metacritic because the critics liked it."

But the critics liked it because it was a good game, right?

I agree that metacritic is not the be-all-end-all source of information about things, but I don't think it's as worthless as you make it seem... it's a good place to go to get an idea for how much you will enjoy a game. Metacritic isn't the only place I go to make decisions about what to buy, but I definitely go there every time.

Quote: Original post by bakanoodle
My feeling is that this number should be nowhere near as important as sales

Of course, sales is the number one priority. It's business after all.

Quote: or your own personal enjoyment out of a game.

Aggregated opinions aren't automatically 'worth' more than a single opinion, but I don't think it's wise to base your business decisions on your own personal preference, other than that it makes sense to stand behind your own work of course.

Quote:
Now. What aspects of the game industry [and other industries] use it for is a completely different issue and the core problem.

Yes exactly. In fact, your whole reply is a point that I failed to get across clearly in my original post. :)

This also depends on the game.

Metacritic focuses on the traditional "hardcore gamer" perspective.

If your game is not in their audience, it will not do well there.

If your game targets the 17-24 year old, it will typically get higher Metacritic ratings.


Consider Wii Play, with a 58 on Metacritic. Yet it is the best selling non-bundled console game in the industry's history. Before people start shouting that it was just because of the controller, look at the rest of the list: Nintendogs (83), Wii Fit (80), Pokemon Red/Blue/Green (82/82/76), and so on. Consider the Sims series, which usually scores in the mid 70's. These are the industry's most profitable titles. None of these are within the hardcore demographics, and consequently they have bad Metacritic ratings.



My last few game titles have been outside that demographic, targeting age 6-10. Our management knows the Metacritic bias, consequently none of the management has been watching the Metacritic scores on the games.

One of the games has a Metacritic score of 66 with comments like "Adults will probably grow bored within an hour", "A bit dull". However, the positive reviews said "If you're over the age of 12, you won't derive the same amount of joy from the game or have the same experience with the title as a child would. In fact, the older you are, chances are the less fascinated you'll be by the game. Kids will enjoy it, and that's what's important." In spite of the 66 Metacritic, we broke two million units in just a few months, and are still selling strong after a year.



Smart businesses understand that critics have their favorite genres, and will know when to ignore them and listen to their customers instead.

theres been a few studies (see articles on gamasutra) that review score has little colleration to actual sales