Hi guys,

Can you give me some advice on running an effective playtesting session? I assume I should give them some sort of document with certain questions about their experience that I would like to know about, and maybe give them a test script as well for them to record any bugs. Well, basically, what things should I prepare for a playtesting session, and what sort of information should I aim to get out of it? And feel free to let me know of anything else I should be thinking about, or any useful resources you have found in the past. Thanks in advance guys, this is really important to me lol >_<

Josh

Josh,

I attended a talk about this very subject. I don't have my notes handy, but I recall these pointers:

- The designer should not be present and visible to the test volunteers. If the test volunteers know they're talking to the game designer, they can soften their remarks. Ideally, the designer could be in a nearby room, monitoring by camera and microphone. It's best if the person running the test session be an uninvolved party so as to elicit the most honest feedback.

- The questions to be asked of the test volunteers should be very focused. Vague questions get vague responses. Know exactly which features you want feedback on, and focus the questions on those.

- Record the test.

- Ask questions before the test, and ask more questions after the test. Let the test volunteers know what you're going to ask them after they've played the game.

- While watching the test, new questions can occur, and those can be passed along to the person running the test.

Good luck with your test.

Indeed, what Tom Sloper said.

You may want to study psychology and social methods for studying, for example your test subjects can't receive help from you AT ALL. If they fail to install the game, you can't help them. If they fail to open the game, you can't help them. If they get stuck on a level, don't interrupt them to give tips. Even if they ask you and insist, you have to say "it's up to you" (but record the question, it's very likely they're confused by something that needs to be addressed). Even if they failed to do something as simple as installing the program or get to the first level from the main menu, there's a 95% chance it's your fault and has to be fixed.

Additionally, you may want to include statistics gathering in your game. For example FPS games record areas where players get killed more often; you can record average life meter, length to pass each level, areas where player spend most/less time, deaths per match/level/etc. Whatever applies to your game.

For example if they spend too much in an area, it may be because it's poorly balanced (an area having too much to explore), players get stuck on something, it's a hard level, its layout looks like there's a hidden/hard to reach area (even though there's none; aka it looks more interesting than it should be), etc, etc... you get the idea.

If you haven't already, I recommend checking out the

If you haven't already, I recommend checking out the

">video by Extra Credits on this very topic.

Great recommendation. One point worth noting: in the video the cartoonist shows the survey question: "rate this game." Don't ask that. Ask "would you buy it" or "how much would you pay" or "would you recommend to your friend/kid brother/grandmother" instead. That's what I meant by "focused" questions. Also: "rate the controls," "rate the graphics," "rate the story" -- all more focused (thus better) than "rate the game." The cartoonist is just using shorthand (it's not a flaw in the video).

First you need to ask a question to yourself, what exatly you want to test? For example testing for bugs is a total waste of testingsession since this can info be gathered easily online by simple "post here all bugs" topic in your forum.

My favourite is to get a person who never played my game and put in front of a computer in me behind his/her backs. I use it for testing interface, confusion and first time experience (the most valuable and hardest to get info). Then I shut up and watch, sometimes I make soft remarks how "unsmart" the player is (which assures I will get a honest opinion without holding anything back due to their politeness at the end :D). I note how much the player cursed/hesitated/got lost on each aprt of the game. At the end I ask "what you think" and I get a list of graviences, sometimes I also ask "what was confusing" (a very good question, since there is no valuating the game, they are telling about their experience not if the game was good or bad, it's comfortable to people). The worst question in my book would be "how you rate the game", very stressful and unlikely to get a honest answer, plus it does not give you much/any info.

Generally, the key is to use the time properly, don't waste it for pointless things you can get by other means. No speeches, no showing documentation, no explaining what the game is about (or, God forbid, introducing the story), except for a very short two line introduction.

Do not forget that playtest session is not to get testing, it is to get specific kind of testing. There are other (much cheaper and easier) ways of testing that should be used for other things.

Ask "would you buy it" or "how much would you pay" or "would you recommend to your friend/kid brother/grandmother" instead. That's what I meant by "focused" questions. Also: "rate the controls," "rate the graphics," "rate the story" -- all more focused (thus better) than "rate the game."

Good point. It's worth mentioning that the wording you use in these kinds of questionnaires is crucial. It will greatly influence the way the tester interprets the question. For example, saying "rate the graphics" is probably a bad idea since the word "graphics" is somewhat ambiguous. It could be interpreted as resolution, textures, use of shaders, or just generally how graphically advanced the game looks. But perhaps that's not what what you meant by graphics - perhaps you wanted the tester to rate how visually appealing the game is, which would include the art style of the game. A better wording may be "rate the visuals". Careful wording of the questionnaire is key to getting the data you want.

If the playtest session is in person, I disagree that the developers shouldn't be present. If you feel that your presence will soften their remarks, they you can leave during any question periods.

I always found the only useful information I would gather from playtests would be from watching other people (from outside the dev team) play the game (no talking). I would see what aspects they used, what aspects were ignored, what areas of the game confused them or frustrated them, which concepts they "got" or didn't "get" (people don't want to feel stupid so they won't admit they don't understand something, it's better to watch for yourself).

You can then use this to ask targetted questions where they can be honest and give useful feedback.

Whenever my team does playtests we always get dozens of useless reports on what each area of the game was rated (if any of these answers is a shock to you then you probably don't know your ass from a hole in the ground), and what "cool features" would make the game better.

I find the value in playtests to be more seeing how an outside user with no experience with the game interacts with it:

• Did they get stuck in the menus (is usability of your interface clear)?
• Did they play a section completely different than you intended? If so, did it frustrate them? Did it expose weird AI? This isn't necessarily a problem (and can be a good thing as it adds variety), but it's worth evaluating your assumptions on why your intended method wasn't chosen (was it not clear they could proceed that way?)
• Is there enough direction to the user (did they get stuck not knowing what to do)?
• Is there too much direction to the user (are they getting frustrated with pop-ups windows or cluttered text)?

If the playtest session is in person, I disagree that the developers shouldn't be present. If you feel that your presence will soften their remarks, they you can leave during any question periods.

We used a fake mirror to have the dev team attend. That works great as only the dev team will notice some very important details.

If the playtest session is in person, I disagree that the developers shouldn't be present. If you feel that your presence will soften their remarks, they you can leave during any question periods.

We used a fake mirror to have the dev team attend. That works great as only the dev team will notice some very important details.

Yes, we use that at our company as well. However the room is laid out such that you can only see two screens from the room behind the mirror, so it's not ideal. Additionally I find that the people behind the mirror often spend more time joking around than paying attention to details (myself included on occassion), although you can put a stop to this by cracking the whip a bit.