Mobile Testing Challenges – Jailbreaking & Rooting

On July 22, 2010 – The U.S. Copyright Office, a division of the Library of Congress, authorized several new exemptions to the Digital Millennium Copyright Act (DMCA), one of which allowed mobile phone users to “jailbreak” their devices to use apps not authorized by the phone’s manufacturer.

While this move was heralded by niche application development shops and iPhone users, Apple was not so thrilled with what they perceived as giving carte blanche freedom to users to “hack” into their tightly controlled Apple devices and add software that may or may not comply with the strict guidelines developed by Apple to protect the end user. Apple typically voids the warranty on iPhones that owners have jailbroken their device. The company maintains that tampering with the iPhone can introduce bugs and glitches.

“Apple’s goal has always been to insure that our customers have a great experience with their iPhone, and we know that jailbreaking can severely degrade the experience,” a company spokeswoman said in response to the Copyright Office ruling. “The vast majority of customers do not jailbreak their iPhones.“

For those users who have Android based devices the practice is known as “rooting”, whereby the user gains administrative access to the root file structure on the device and then can modify the operating system and add applications that take full advantage of root level access to the OS.

There are a multitude of websites dedicated to the tips and tricks for rooting and jailbreaking a mobile device.  Some clearly define the potential risks of moving away from the manufacturer’s standards. Others fail to appreciate or understand the potential pitfalls of ‘hacking” into the operating system of the device.  They extol the virtues of personal freedom, and ignore the potential risks to the user, that they are largely unaware of.  In his March 22, 2011 blog on androidauthority.com titled Rooting for dummies: A beginner’s guide to rooting your Android device by Derek Scott on Mar 22, 2011, Derek Scott speaks to the risk of rooting:

“Rooting your phone does come with some risks. The most notable risk is that you will void any warranty that you have on your device…Other than voiding your warranty, there isn’t that much risk involved. Some users occasionally run into problems, the most nefarious of which is bricking your device (i.e., rendering your phone completely non-functional–pretty much like a brick). Other possibilities include bootlooping, in which your phone boots, reboots, boots, reboots, in an endless cycle. The chances of running into such problems, however, are very slim–provided, that you follow instructions properly.”

While Derek clearly understands the risks, he fails to mention all of the risks that root access exposes to the device owner, such as the potential security threats from applications that haven’t been through a rigorous review process.  His perspective is taken from the perspective of damage the person who does the rooting can do, rather than to understand that the most intense risks come from the outside.  This is either naivety on his part or a purposeful agenda to ignore the obvious.

Testing in a Mobile World

So how does jailbreaking and rooting impact the testing world?  There are many automated mobile testing solutions on the market.  As you dig into the mechanics of each of the offerings, you quickly find that most require the device under test to be jailbroken or rooted in order to conduct the automated testing.  The vendors don’t offer that information up front as you sit through the polished demo of the solution, but through a bit of questioning they will admit that they do, in fact, require the device to be hacked.  The vendor will quickly assure you that rooting or jailbreaking the device has no adverse impact on the testing and that the results of the testing are above reproach and completely valid.

Another option is to select a vendor that does not require the device to be rooted or jailbroken.  These vendors typically will have a component that is compiled with the solution and is a permanent part of the application, thus removing the need to root or jailbreak the device.

There are multiple potential issues when looking at the testing solutions that require the device to either be rooted or jailbroken.  The company that is considering a mobile testing solution should consider those potential issues while evaluating the solution to ensure that they do not adversely impact the organization.  Let’s look at just a few of the issues related to mobile testing on a rooted or jailbroken device

Security – When an iPhone is jailbroken a large percentage of the built in security of the OS is disabled. This means that the device is now at risk.  While in the testing organization, most of your devices are not going to contain sensitive data, such as, personal information, the potential for malicious software to get on the device increases.  The 2010 Copyright Office ruling was a green light for worldwide hackers to start focusing on the iPhone device as a vehicle for malicious attacks. Rooting presents the same risk as the device’s administrative directory system is now exposed. Using a jailbroken or rooted device may violate company security policies and is a risk that is just not worth it considering there are other options.

Device Updates – Apple has gone to great lengths to protect their operating system from unintended consequences.  Jailbreaking is in direct conflict with the protections that Apple has put in place.  Apple takes every opportunity to remedy this in the way in which they apply OS upgrades.  When the user plugs their mobile device into their computer to update it, Apple closes the operating system before installing the new update.  This process ensures that the “hole” is closed on an update of the device.  Testing organizations should consider the impact of manipulating the operating system of the device over time.  With each OS upgrade or patch, the device would need to be jailbroken again.  This constant manipulation of the OS in a manner not intended by the manufacturer may result in the device becoming unstable, and unreliable for testing.

Test Like You Fly – When I was in aerospace we experienced a flight issue on a Titan rocket where the vehicle mission failed due to an issue with the software.  In our infinite wisdom, we had developed a very generic set of ground software, which was designed to ensure the vehicle was ready for launch.  This Titan vehicle had been built to handle multiple payloads, and so the ground software was developed to test that both payload connectors correctly separated the payload from the vehicle.  To save effort, the ground software was developed to send the separation signal to both the forward and aft payload connectors at the same time.  The vehicle was assembled at the launch pad; the payload was installed; and the ground test verified that the payload was, in fact, getting a separation command.  The launch was a “go”.

The Titan hurled its way into space in what everyone considered a near perfect fashion.  The Stage 0 solid boosters performed flawlessly and separated from the vehicle while still in the atmosphere and fell into the ocean.  The Stage I rocket pitched and yawed the vehicle out of the Earth’s atmosphere where the three piece payload faring separated like a beautiful flower opening on a spring morning.  Stage II kicked in and the single main engine guided the vehicle into perfect position to place the commercial satellite perfectly into place in orbit.  The guidance software issued the separate command to the payload.  On the ground we anxiously awaited the telemetry signal back from the vehicle that indicated that the payload had, in fact, separated and we could claim yet another success in a long line of launches over the nearly 4 decades of launches…Nothing.  No signal ever came from the vehicle.  Mission crews scrambled to find out why.  They tried manual overrides.  They tried resetting the software.  They tried it all, but no signal ever came.  The orbit of the vehicle with its payload still attached eventually decayed and the vehicle and its precious cargo burnt up on reentry.

 A major disaster

Work began immediately to understand what could have possibly gone wrong. Every possible piece of data was reviewed.  All of the telemetry from the vehicle was played and replayed to try and understand what possibly could have happened.  After months of reviews the findings were published.  It was determined that when the vehicle was stacked at the launch pad and the payload installed, the forward payload connector was connected to the payload, even though the payload was in the aft payload position.  This in and of itself would not be an issue, but the flight software was designed to only issue the aft payload command because the vehicle only had a single payload.  When the flight software issued the separate command the aft payload connector received the command, but there wasn’t a payload attached to it.  This too of itself would not have been a problem had the ground software that verified the vehicle not been developed to issue the payload separate command to both payload connectors at the same time.  Since the command was issued on the pad to both connectors, the ground software received a payload separate response from the vehicle.  It believed all was right with the vehicle and cleared it for launch.  This vehicle failure took three separate errors that by themselves could have been recovered, but together spelled disaster for this launch.

The lesson learned was that from that day forward all testing would be done in a way that absolutely replicated the flight configuration of the vehicle. There would be no tolerance for cutting corners or simplifying testing work.  The phrase Test Like You Fly was born.  The premise being that anything you do to change your environment from what the actual operating environment will be like adds unacceptable risk to your testing.

Now I know some of you are thinking to yourselves that your applications aren’t critical like a launch vehicle.  Some may have even used my most hated phrase, “relax, this isn’t rocket science”.  But before you completely dismiss the premise, let’s consider how critical your mobile apps are to the success of your business.  If your mobile application is a banking app, it probably is pretty important that it function securely and correctly for your customers.  The test like you fly premise says ensure that the operating system and application combination (launch vehicle and payload) are tested in the way they will operate in the “real world”.  When you root or jailbreak a device you are taking a shortcut to testing and putting the device into a state that doesn’t match how most users will use the system.  This is much like writing the ground testing routine to issue the command to both forward and aft payload connectors at the same time.  While it can definitely tell you that neither connector is connected, it cannot correctly tell you that the vehicle and payload package is configured correctly.  Testing in and of itself is a risk avoidance strategy.  Do it correctly and you prevent disaster from occurring once your application is released.  Cut corners and you reduce the effectiveness and ultimate value of the testing activities.  If your applications aren’t mission critical to your business why bother even building them.  Last time I checked revenue was always mission critical!

Jailbreaking and rooting may have its place in public consumption, but it is not the most effective way to ensure you are effectively testing the mobile OS and application package.  When you are assessing which mobile testing solution works for your organization, keep the test like you fly mantra in mind.  In fact, we should change it to something more mobile, like Test Like You Text.  Happy testing!