You are almost there. The finish line is so close you can taste the champagne toast that comes with victory and a job well done. But … but, it is just not working. There is a bug in the app and the development team cannot figure out where it is and how it has tossed a wrench in the entire process. Progress has come to a complete stop and the finish line, once so close, might as well be a thousand miles away.
Mobile app testing is not easy. Whenever I talk to a developer, testing inevitably is one of the phases of creation that is both exciting and excruciating – exciting because a product is finally near completion, excruciating because… well, if you have ever tried to dig unknown bugs out of software, you understand. To do testing right, it is a multi-layered approach that takes time, resources and patience. There are several ways to go about app testing and not all situations make sense for every developer.
Multi-Layered Testing
We enlisted IBM’s Leigh Williamson, a Distinguished Engineer and a member of the CTO Team, to guide us through the different ways that developers can go about app testing. Testing is not as simple as a developer trying to troubleshoot the native code on an app. Mobile apps are often vertical software systems with a variety of moving parts on the front end, in the middle and in a back-end cloud. Your code on the device may be running perfectly fine, but you would never know it because it is being corrupted from the back end that feeds it information. Or maybe some of the middle-tier services, like third-party SDKs, are running improperly. When something goes wrong, sometimes it is easy to figure out what is broken. Many times, you have no idea.
There are a variety of ways to automate testing. The most painful way to do it but probably one of the most essential is the manual test. Put the code on the device and use it like you would in the real world. When something goes wrong, log it and dig back through the code.
“There is manual testing for mobile applications. Even though that is the most inefficient and costly, we do still think there is a place for it in a well-balanced strategy. Because we really haven’t gotten a way yet that we’ve seen that can automate the valuation of the usability and consumability of the general end-user experience for the mobile app. That is something that a real human being has to evaluate ultimately,” Williamson said.
In an ideal world, manual testing is one of the last steps in the application development lifecycle. Everything has been built and it is almost ready to ship. There are ways to make it less painful, such as services from startups like Boston-based Crashlytics that provides a SDK that will tell you why your app crashed to the precise line of code along with all the environmental data at the time of crash.
Is That Cloud Really a Wrench in Your Machine?
As mobile app developers turn to the cloud, there is then another tier that needs to be tested. In IBM’s mind, this is where its strength in the testing realm lays. The company can simulate or isolate the middle and back-end tiers to give developers an idea of how the app will perform before making actual integrations between device, servers and clouds. This is function of IBM’s acquisition of Green Hat, a testing platform for a variety of cloud services. A “back-end as a service” mobile cloud service like Kinvey will give developers an instance of Kinvey’s cloud structure (that runs through RackSpace, Azure and Amazon) that runs on the desktop to perform a similar service.
“What that allows an organization to do is to actually set up a continuous test agile methodology for developing the mobile application, checking in some code change which triggers in an automatic build of the mobile binaries for the app for the different mobile devices, and push the updated version of the application to some real mobile devices, and execute the code on the mobile devices with a simulated middle tier and back-end tier because it is always going to respond to the mobile device the same,” Williamson said.
A Cloud of a Different Sort
A developer recently lamented to me about testing, and his main comments were along the lines of “there are too many mobile devices out there running too many damned versions of operating systems across too many mobile carriers. It is freaking impossible to test all of it.” That was the gist of his complaint. The actual words were … saltier.
He has a point. There are upwards of 300 different types of Android smartphones on the market across the world. There is an increasing amount of Windows Phones from different carriers on the market, not to mention developers that might want to run applications or mobile websites on BREW, Symbian or Bada devices. Even the iPhone has become more difficult to test applications on with three different levels in the market (3GS, 4, 4S) that most consumers use. There are also different levels of iOS (not everybody knows how or cares to update) running on different carriers. iOS is not fragmented like Android is, but there are still variables to be tested within the ecosystem.
To test different devices, a developer could acquire all those smartphones and tablets, rack them up on a server, and pound away on each device specification. That is impractical, time consuming and expensive. There are several services that developers can turn to for a “device cloud” that developers can use to test applications on many devices at once. IBM uses Perfecta Mobile and Device Anywhere for its device clouds but there are also solutions from companies like Apkudo for device-cloud testing.
“That is also part of our strategy, recognizing that there are many different techniques available for mobile testing and some of those techniques are applicable for mobile apps but for other reasons customers would find the technique not acceptable,” Williamson said.
Choices … Or, None at All?
IBM has been around for a long time – more than 100 years, in fact. It was one of the founding companies in the computing space and to develop consumer hardware and later make the switch to enterprise software. There is chutzpah to being IBM. They take a deep look at issues facing developers and create verticals to solve those problems. And charge for it, sometimes handsomely.
There are benefits to working with IBM. For instance, there are not many other companies out there that offer the end-to-end testing services that IBM can offer. If the company does not have the solution itself, it will either acquire the solution (Green Hat) or partner with a company that does (Device Anywhere). As for Williamson, he understands that end-to-end testing can be done without the services of IBM, but it requires a lot more work than perhaps necessary.
“There are other alternative ways to reach that same well-integrated solution but it requires the customer to do a lot of the integration themselves and in essence be a systems integrator of their own whereas we do that in IBM as a part of our mission,” Williamson said.
Bug and cloud keyboard images courtesy of Shutterstock