Android Multi-Devices Remote Testing

Background

Robotium is an Android testing framework that provides full support for Activities, Dialogs, Toasts, Menus and Context Menus in order to instrumenting user interface integration tests. Due to Android devices fragmentation in the market, testing apps on the simulator has limited bug reproducibility. In addition, testing on individual devices one at a time creates many tedious setup steps and manual work for mobile developers.

The Idea

The objective was to experiment and take the local testing functionality further provide the capability of instrumenting test suite over the network creating a client/server architecture.

Thus, Remote Robotium was created as an extension to the current Robotium framework. It allows us to test multiple emulators or even real device instances at the same time with JUnit(without Android JUnit), or even execute remote instructions without any test frameworks.

Key Objectives

I would like to make sure the following objectives can be achieved in the implementation:

• It should maintain existing Robotium compatibilities for single device (aka. solo mode) testing.
• Developers have the option to opt-in for remote testing without large refactoring.
• Maintain all the UI component testing capabilities in remote mode.
• Testing framework on Android devices was limited to use Android JUnit, it would be nice to remove such dependency and allow the use of standard JUnit or any preferred instrumentation frameworks.
• Maximize the number of devices can be tested over the network while the local network has limited throughput for stringified test instruction and test result.

Implementation

There are several challenges discovered during the implementation phase. I need to find a way of efficiently translating local in-memory test instructions to network-based instructions, with these constraints, I still need to maintain the local testing functionality. At the same time, I should allow users to enable remote testing by adding one single additional flag (i.e. client.remote = true).

There were tons of refactoring for creating a new interface for both SoloClient and RemoteClient to ensure common instructions are implemented downstream. Then I utilize Java’s reflection to translate method signatures and parameters to a Remote Procedure Call. I also implemented a server to be installed on the device once. The server is capable of receiving, decoding the RPC and perform the action, and more importantly, provide executions summary for reporting back the results.

The second major challenge is error handling. The typical types of errors are device timing out, networking timeout, testing behavior inconsistent or simply devices incompatibility. Testing behavior inconsistency is important to produce meaningful and traceable results for the tester to really diagnoses the issue.

Demonstration

The recorded video shows:

• A standard JUnit test suite was able to launch test instances in three simulators and one additional real device, four in total.
• Devices are all connected to the same local network.
• Various click, rotation, and text entries tests were executed on all four instances.
• The JUnit test was able to report the test suite summary as usual.

Example SDK Test Case

This is an example of testing inconsistency and expecting to produce a test assertion failure-note 1 and note 2 content are expected to be different.