A recent report by OpenSignal revealed that there are 18,796 distinct Android devices in use. This data was gathered directly from users who installed the company’s app. Even though device fragmentation is not a new phenomenon in the Android world, the sheer number of devices is impressive (up from 11,868 devices counted last year).

OpenSignal’s data might not reveal the full picture, but it is fairly representative of the Android ecosystem and clearly illustrates what mobile application developers have to deal with. And yet the myriad of different hardware (and thus screen sizes, CPUs, memory sizes, sensors, etc.) is not the only challenge for application developers. The report also shows that there are seven versions of Android in use with market shares between 0.7% (Android 2.2) and 26.5% (Android 4.1). The latest version of Android (4.4) was installed on 20.9% of surveyed devices. OpenSignal states that the Android operating system is the most fragmented it has ever been.

What does this mean for users and developers?

The app stores are full of reviews from users with complaints that an app is not running, is crashing, or is hanging on their particular device. Almost always, these users give the app the minimum possible rating (sometimes along with a nasty comment). And who could blame them? An otherwise great app that does not work on someone’s device is of no use for them. Companies put their reputation at risk when they release “half-baked” apps or faulty updates. In almost all cases, user frustration could have been avoided if a proper testing strategy was in place.

While it seems obvious that an app's success depends on its type and its features, the real difference between an app that is successful and one that isn't, is testing. Users are more likely to abandon an app if it doesn't function properly.

How to test (hybrid) mobile apps

“Testing” is a general term that can be broken down to specific types of testing. Some of the most common are:

  • Unit testing relies on automated tests written by developers during the development process. Each test targets individual units of source code or a narrow aspect of application behaviour. For more information on unit testing, see Effective Unit Testing.
  • Functional testing is performed by QA personnel or through automated UI testing frameworks.
  • Performance testing is done manually with profiling tools or through integrating app analytics to gather data from real usage.

The right testing approach depends on several factors, but most of all, it has to match the overall app development method. As laid out in Building Cross-Platform Mobile Apps, there are three types of mobile applications: native, web-based and hybrid.

  • Native apps are built for a specific platform (e.g., iOS, Android, Windows Phone) using the platform’s native SDK. If you take this approach, you can use the native testing tools that come with the corresponding development environment. However, unit tests, as well as any scripted UI automation tests, are not interchangeable and will only work on that particular platform.
  • Web applications are applications that are built using web technologies such as HTML5, CSS and JavaScript. For testing web applications, you can rely on a large number of established testing frameworks such as Selenium for browser automation. Unit testing can be performed with tools like Jasmine and JsUnit.
  • Hybrid apps combine the two aforementioned approaches. A hybrid app consists of a web-based component and native code. This combination allows reuse of large portions of code and can significantly reduce the effort of targeting several platforms. On the other hand, comprehensive testing of hybrid apps can be more demanding.

The hybrid approach has several advantages and is becoming increasingly popular. Frameworks for mobile cross-platform development such as Sencha Touch (in combination with PhoneGap) or Xamarin have significantly matured over the last few years. Due to a shared codebase, they allow cross-platform unit testing. When it comes to functional testing, these frameworks do not always offer the same level of quality as native tools. Developers, therefore, often choose native testing solutions, having to implement platform-specific test scripts.

A cross-platform testing framework

A tool that aims at closing this gap is Appium. Appium is an open source framework for test automation. It allows developers to write tests against multiple platforms (currently iOS and Android), using the same API. This enables code reuse between test-suites. Appium supports both native and hybrid apps. Under the hood, Appium uses vendor-provided automation frameworks (UIAutomation for iOS, UiAutomator for Android 4.2+ and Instrumentation for Android 2.3+). That way, the app to be tested does not need to be recompiled or modified to include any third-party code. The app you are testing is the same app you are deploying.

The Appium API is a version of the WebDriver API, extended by methods for mobile automation. WebDriver (aka "Selenium WebDriver") has become the de facto standard for automating web browsers, and is a W3C Working Draft. With this client-server architecture, test clients written in any language can be used to send the appropriate commands to the Appium server. There are client libraries in Java, Ruby, Python, PHP, JavaScript, and C# which support Appium's extensions to the WebDriver protocol.

Appium is basically a webserver that exposes a REST API. It receives connections from a client, listens for commands, executes those commands on a mobile device, and responds with an HTTP response representing the result of the command execution.

A test-suite consists of a sequence of Appium commands that allow finding UI elements in the app and performing UI interactions. A command is always performed within a context. If you are testing a hybrid app, the context determines if the command is executed in the native or the web-based part of the app. Seamless switching between contexts allows comprehensive cross-platform testing of hybrid apps with just one framework.


Testing of mobile apps can be tedious and challenging, especially in the highly fragmented Android world. Automated testing with emulators and real devices will help remove bugs earlier in the development life cycle. With a comprehensive testing strategy, changes to the app and adding new features can be done without the fear of breaking existing functionality.