Mobile Application Testing: Should The Device Matter?

At the STAR EAST conference this year, mobile solutions saturated the vendor exhibition area. It’s no wonder that mobile application testing is the hot in the testing community right now. Testers are now faced with the task of testing mobile versions of their applications and figuring out how to adapt to the new paradigm (and challenges) mobile applications bring. It’s not as if they didn’t have enough to test already, right?

After listening to some of the conversations around mobile and seeing some interesting questions being raised, I’m wondering if this has become more difficult than it should be. A common question I hear is “How do I figure out how to test all of these different devices with various screen sizes by different manufacturers with a different OS on each?”  A quick look at the market reveals at least four major operating systems (Apple, Android, Windows, and Blackberry), with multiple versions for an untold number of devices for each.  I think every tester would agree, automation is a must.

So, should the device matter? For years we have been testing applications on various laptops, desktops, and servers without regard to the model, size, or parts inside. Aren’t we just replacing the PC with a mobile device? The reason I could see the specific device playing a key role today has to do with the way mobile automation is being accomplished – namely through bitmap recognition and/or OCR technologies. These technologies are useful, and serve their purpose for building automated test scripts. However, when screen sizes change (i.e. iPhone to iPad), this breaks the scripts. Worse case, multiple versions of the script need to be in place to accommodate different devices.

As an HP reseller, we’re focused on the automation capabilities of HP’s Unified Functional Test, which includes QuickTest Pro and Service Test. There are a few add-ins that can be purchased to add mobile automation. They include ZAP-fiX, DeviceAnywhere, and Perfecto Mobile. All of these handle automation using bitmap recognition and/or OCR. They also currently require the device to be jailbroken (iPhone) or rooted (Android) to start testing. This causes a major dilemma for the QA Manager committed to quality. How can you start with a fundamental change to the OS code? It’s a hack, which means it is an undocumented change to the release being tested that does not reflect the production end user build (unless they have the exact jailbreak too) with no control over that changed code. Nevertheless, testing must get done.

UPDATE 02/07/2013 – Perfecto has announced that they no longer require jailbreaking of the devices: Perfecto Mobile Introduces Full-Featured, Non-rooted iOS Mobile Testing in the Cloud. However, as of this writing there is still no recognition of objects beyond the OCR level of recognition. See our post about the top mobile automation tool requirements to read about about the differences in object recognition technologies.

JAMO Solutions provides code that is integrated into the application as it is delivered to the device. This code allows for object recognition so that an automation product like QuickTest Pro can “hook” into the application and see all of the objects, their name, and many other properties. Why is that important? Now we have access to all the objects of the application various known properties that bitmap recognition methods could never be aware of (like hidden objects that might still need some form of verification). Scripts won’t break due to screen size because the object’s property is still the same. This means the OS matters, but the device doesn’t. If the device supports the OS, then the script will still work even when the device changes. This is how testers currently approach web based application test automation (non mobile). Testers are not concerned whether they are testing against a Dell desktop, a Lenovo laptop, or an HP Proliant server. They obviously do care whether the OS is Windows 2008, Windows XP, or Red Hat Linux.  However, if the OS supports the test case, that is what matters.

Northway Solutions Group is partnering with multiple mobile vendors, including JAMO, Mobile Labs, Perfecto, and others. We provide the role of the trusted advisor to our clients, and provide them with the solution that best fits their needs.  When we talk to our existing customers who already use HP’s QuickTest Professional, we can help them understand how to extend its use into mobile applications that is best for them.

What do you think? Should the device matter? Why or why not? We look forward to your comments.