Speaking of mobile game testing, it is a common misconception that manual testing is the only way to go forward. In many cases it is the first thing to start with, but to really get all issues spotted out and fixed before the game is published, test automation built into the process, hammering each and every regression and advancement can yield significant results when considering the customer-ready game.

Let's face the fact: manual testing has still too many disadvantages as it's way too time-consuming, tedious, error-prone and not being able to thoroughly, systematically do the testing of whole software entity. Manual testing is somewhat like scratching the surface and you can't really find out what's going on underneath the UI. Besides, you would need an army of programming-capable guys to debug all bits and pieces - something that test automation can deliver you 24/7 - without any manual efforts.

For reason or another, some people still associate manual testing as the only way to test games. Sure, there are business reason for many companies out there to emphasize it as they are eager to sell their services (or crowdsource it to unknown quasi-testers) but again, it is the technology and test automation that can get in depth and make sure app works across the ecosystem of different devices, with different software setups, hardware configurations and so on.

There are lots of differences between test automation frameworks and the pros and cons of using different test automation frameworks for testing . One great example and very useful framework is Appium - and new advanced features, such as image comparison and recognition.

UI and Functional Testing - with Appium

In a nutshell, Appium is a mobile UI testing framework supporting cross-platform testing of native, hybrid and mobile-web apps for iOS and Android. In fact, Appium is a pretty good choice for mobile games as in many cases those games tend to be identical - or at least very similar - on both platforms, Android and iOS - and the same test script can apply for both. But there are also some other reasons why Appium is a great choice for mobile game testing. For example, you can write tests using your favorite development tools/environment and programming languages, such as Java, Objective-C, Javascript, PHP, Ruby, Python, C# and so on.

Appium enables you to execute your tests on mobile device irrespective of the device OS. This is because the framework is basically a wrapper that translates Selenium Webdriver commands into UIAutomation (iOS) or UIAutomator (Android, API level>=17) or Selendroid (Android, API level <=16) commands depending on the device type. For example, in the context of Android, this is how Appium compares to other test automation frameworks:

EXAMPLE: Using Appium to test Clash of Clans (game by Supercell)

In this example we are using Supercell's Clash of Clans game. A fantastic game and I bet many of you have played it so you should be pretty familiar how the game looks and so on. We're also going to use Appium as a selected test automation framework to basic clicking-through of Clash of Clans tutorial.

##
## Example script that tests Clash of Clans tutorial first steps
##
## Works on different resolutions, both iOS and Android
##
 
import unittest
from time import sleep
from TestdroidAppiumTest import TestdroidAppiumTest, log
from selenium.common.exceptions import WebDriverException
 
class ClashOfClansTest(TestdroidAppiumTest):
    def setUp(self):
        # TestdroidAppiumTest takes settings (local or cloud) from environment variables
        super(ClashOfClansTest, self).setUp()
 
    def test_tutorial(self):
        driver = self.get_driver() # Initialize Appium connection to device
 
        sleep(10) # Wait that game loads
 
        # Use this to get detected screen hierarchy
        # print self.driver.page_source
        # Dismiss the in-app purchases dialog if it shows
 
        okayButton = None
        if self.isAndroid():
            try:
                okayButton = driver.find_element_by_id('button3')
                okayButton.click()
                sleep(5)
            except WebDriverException:
                log("There was no in-app purchases dialog")
        else: # iOS
            self.driver.implicitly_wait(5)          # wait only 5 seconds to find it
            try:
                okayButton = driver.find_element_by_accessibility_id('Okay')
                okayButton.click()
                # No need to sleep here since for iOS we wait the Game Center to popup...
            except WebDriverException:
                log("There was no in-app purchases dialog")
            self.driver.implicitly_wait(30)
 
        # Cancel iOS Game Center login
        if self.isIOS():
            #print self.driver.page_source
            try:
                self.driver.implicitly_wait(5)
                cancelButton = driver.find_element_by_accessibility_id('Cancel')
                log("Canceling iOS Game Center login...")
                cancelButton.click()
                sleep(2)
            except WebDriverException:
                log("The Game Center login was not displayed")
            self.driver.implicitly_wait(30)
 
        self.screenshot("welcome-chief")
 
        # Check that there is a goldmine on screen
        rect = self.find_image("queryimages/tutorial_goldmine.png", screenshot_match="screenshots/goldmine_match.png")
        self.assertIsNotNone(rect, "There should be a goldmine on screen in beginning of tutorial")
        log('Gold mine found at %s %s! Tapping tutorial forward...' % (rect[0], rect[1]))
 
        # Dismiss the bubbles
        self.tap_middle()
        sleep(2) # second blabla
        self.tap_middle()
        sleep(2) # Goblin appears
        self.tap_middle()
        sleep(1)
 
        # Go to shop
        # NOTE: tap_image does also assert, fails test if target not recognized
        self.tap_image("queryimages/shopbutton.png")
        sleep(1)
 
        # Buy cannon
        self.screenshot('cannon')
        self.tap_image("queryimages/cannon.png")
        sleep(2)
 
        # Place the cannon
        self.screenshot('place_the_cannon')
        self.tap_image("queryimages/place_the_cannon.png", width_modifier=0.75)
        sleep(2)
        self.screenshot('finish_now')
        # Use gem to finish right away
        self.tap_image("queryimages/finish_now.png")
        sleep(3)
        # Bring it on!
        self.screenshot('bring_it_on')
        self.tap_image("queryimages/bring_it_on.png", height_modifier=0.75)
        sleep(10)
        self.screenshot('battle')
        sleep(10)
        self.screenshot('end_of_battle')
 
        # To be continued...
 
if __name__ == '__main__':
    unittest.main()

Let's look some stages in this script. The test_tutorial contains the following steps:

1. It first figures out if test is executed either on Android (self.isAndroid()) or iOS. As you can see, it looks content different, on Android it is trying to find by element ID and on iOS by accessibility ID with description ('Okay'). The same check happens for iOS Game Center login.

2. Screenshots are taken in various steps and stored in files entered as a parameter in a function call.

3. We do a check if "goldmine" exists on screen by comparing two pngs using self.find_image call. If these pictures match (=goldmine exists on screen), we'll go forward with tutorial.

4. We proceed with the tutorial with the following steps: 1) Go to shop, 2) Buy cannon, 3) Place the cannon. The information about all these three items is stored in those pngs: shopbutton.png, cannon.png, place_the_cannon.png.

5. Finally, we finish the tutorial and start the battle! After battle, application is brought down.

Okay. That's about the programming and scripting for now and let's see how that script looks on real devices. We used one iOS (iPhone 4S) and two Android phones (Samsung Galaxy S3 Mini and HTC One X) for this script. The video footage is taken at our office and is not a high-quality but you should see something going there on devices:

How we are using Image Recognition?

What we just went through in Appium+Clash of Clans example was the basic image recognition flow for enabling mobile game to be tested on real devices, regardless OS was different (Android and iOS). There are different ways to recognize content from images. For example, template matching is a technique for finding small parts of an image which match a template image. It is actually very handy even for recognizing UI elements and graphics resized and/or rotated in different form. This is one way how some Testdroid game developers use image recognition today.

Let's say the template image has some strong features - e.g. text easy to be abstracted from the background content - a feature-based approach can be used. In this example "Button 1" text was resized and rotated (or if it is otherwise transformed) this can be quickly and easily identified and further actions can be taken. Happy Testing!