Mobile Application Testing: Should The Device Matter?

At the STAR EAST conference this year, mobile solutions saturated the vendor exhibition area. It’s no wonder that mobile application testing is the hot in the testing community right now. Testers are now faced with the task of testing mobile versions of their applications and figuring out how to adapt to the new paradigm (and challenges) mobile applications bring. It’s not as if they didn’t have enough to test already, right?

After listening to some of the conversations around mobile and seeing some interesting questions being raised, I’m wondering if this has become more difficult than it should be. A common question I hear is “How do I figure out how to test all of these different devices with various screen sizes by different manufacturers with a different OS on each?”  A quick look at the market reveals at least four major operating systems (Apple, Android, Windows, and Blackberry), with multiple versions for an untold number of devices for each.  I think every tester would agree, automation is a must.

So, should the device matter? For years we have been testing applications on various laptops, desktops, and servers without regard to the model, size, or parts inside. Aren’t we just replacing the PC with a mobile device? The reason I could see the specific device playing a key role today has to do with the way mobile automation is being accomplished – namely through bitmap recognition and/or OCR technologies. These technologies are useful, and serve their purpose for building automated test scripts. However, when screen sizes change (i.e. iPhone to iPad), this breaks the scripts. Worse case, multiple versions of the script need to be in place to accommodate different devices.

As an HP reseller, we’re focused on the automation capabilities of HP’s Unified Functional Test, which includes QuickTest Pro and Service Test. There are a few add-ins that can be purchased to add mobile automation. They include ZAP-fiX, DeviceAnywhere, and Perfecto Mobile. All of these handle automation using bitmap recognition and/or OCR. They also currently require the device to be jailbroken (iPhone) or rooted (Android) to start testing. This causes a major dilemma for the QA Manager committed to quality. How can you start with a fundamental change to the OS code? It’s a hack, which means it is an undocumented change to the release being tested that does not reflect the production end user build (unless they have the exact jailbreak too) with no control over that changed code. Nevertheless, testing must get done.

UPDATE 02/07/2013 – Perfecto has announced that they no longer require jailbreaking of the devices: Perfecto Mobile Introduces Full-Featured, Non-rooted iOS Mobile Testing in the Cloud. However, as of this writing there is still no recognition of objects beyond the OCR level of recognition. See our post about the top mobile automation tool requirements to read about about the differences in object recognition technologies.

JAMO Solutions provides code that is integrated into the application as it is delivered to the device. This code allows for object recognition so that an automation product like QuickTest Pro can “hook” into the application and see all of the objects, their name, and many other properties. Why is that important? Now we have access to all the objects of the application various known properties that bitmap recognition methods could never be aware of (like hidden objects that might still need some form of verification). Scripts won’t break due to screen size because the object’s property is still the same. This means the OS matters, but the device doesn’t. If the device supports the OS, then the script will still work even when the device changes. This is how testers currently approach web based application test automation (non mobile). Testers are not concerned whether they are testing against a Dell desktop, a Lenovo laptop, or an HP Proliant server. They obviously do care whether the OS is Windows 2008, Windows XP, or Red Hat Linux.  However, if the OS supports the test case, that is what matters.

Northway Solutions Group is partnering with multiple mobile vendors, including JAMO, Mobile Labs, Perfecto, and others. We provide the role of the trusted advisor to our clients, and provide them with the solution that best fits their needs.  When we talk to our existing customers who already use HP’s QuickTest Professional, we can help them understand how to extend its use into mobile applications that is best for them.

What do you think? Should the device matter? Why or why not? We look forward to your comments.

What's Next?

Did you enjoy this article? Help spread the word by sharing:

Join the Northway Navigator Club today and get access to restricted content including our best tips and tricks. Membership is free! You will also receive free email updates by registering.

Engage in the conversation and leave a comment:

Scott Moore

About Scott Moore (153 articles)

With over 20 years of IT experience with various platforms and technologies, Scott has tested some of the largest applications and infrastructures in the world. He is a Certified Instructor and Certified Product Consultant in HP’s LoadRunner and Performance Center products. He currently holds HP certifications for ASE, ASC, and CI. A thought leader in the APM space, he speaks regularly at IT conferences and events



  • Alex Weiss

    The premise statement “we never cared about the PC type and it’s insides” is wrong. We did. Always. Not for the automation of the load, but for automation of the functional – we did. It’s depended on the type of the tested app how much we cared about the testing PC. Same is with apps on mobile. The reason item care more about the device is because same OS behaves differently on different devices and moreover in Android which is strongly varying across different devices. I think Win and iOS devices are less pain to test because variance is minimal on them. My 2 cents.

    • http://northwaysolutions.com/ Brian MacKenzie

      You’re right – the device does matter (to a degree). The Android Market allows developers to compile targeted builds for specific devices. If you are already creating targeted builds, then the device obviously matters. Otherwise, you may not find the need the test individual devices that operate within the same OS.

      If you do need to test specific devices, this is where M-eux (Jamo) excels by utilizing object based recognition. A single QTP script can be data-driven so that multiple iterations can be executed, one for each device, and the results can be captured in a single report. For example, execute one iteration on the HTC Evo and another on the Galaxy Nexus.

  • http://www.northwaysolutions.com/ Scott Moore

    My statement was “For years we have been testing applications on various laptops,
    desktops, and servers without regard to the model, size, or parts
    inside.” From an application functional testing perspective this is true, but in the 80′s application automation solutions did require the equivalent of “agents” or additional code to make the testing tool work. This bothered developers back then as it does for mobile developers today. I am not saying we don’t care about the hardware components, but that is another kind of testing. Strictly from an application layer functional testing viewpoint, how is changing the power supply brand or memory or NIC card relevant? I’m asking the question. It’s quite possible (however unlikely), that I, the great Loadtester, could be wrrrrrrrr… not considering all facets of this….. LOL!