How to Test Mobile SDK

Traditional SDK (Software Development Kit) is typically a set of software development tools that allow the creation of applications for a certain software package, software framework, hardware platform, computer system, video game console, operating system, or similar development platform. In mobile and big data days, various SDKs play more and more important roles in multiple domains, including payment, customer services, background operation, ad exchange, push, track, cloud storage, test analysis report, big data analysis, etc. Therefore, SDK testing is gaining more attention. One of the most popular usages of SDK in mobile world is to collect user behavior data for big data analysis thus provide solid basis and guidelines for business strategies. Taking this kind of SDK as an example, we’d like to share some ideas and experiences on how to test mobile SDKs.

The exist criteria for testing this kind of SDK are from below three aspects:

1. Stability

• App with SDK integrated must work correctly. All features function as they are.
• App performance is still good enough after the SDK is integrated.

2. Accuracy

• The target data is collected correctly.
• The target data is transferred and stored to backend database correctly.

3. Security: SDK integrated into app will not expose extra vulnerability to the app.

To make sure of meeting above criteria, a lot of investigation is necessary before implementing the app. Also a good design is the key, including but not limited to: how to isolate the workflow and logic of SDK so that it can adapt to different apps developed by different teams; how to improve SDK tolerance so that the error from SDK itself will not affect the app; how to collect, transfer and store data.

From testing point of view, the validation on business workflow of the app is not the focus, however, testers still need to know the app well, because main workflows still need to be gone through to make sure they work well after the SDK is integrated and as many as possible UI elements should be gone through to make sure that user behavior on them is captured and counted correctly.

Let’s look into more details.

Stability

1. Functionality: Go through main workflows to make sure they work well. If there are errors, compare with the behavior of original app without the SDK to clarify if it’s caused by SDK or not.

The bugs in SDK often cause mal-functionality or even crash of the app. For examples, suppose that the max size of one collection of data is 49 bytes. If only consider most common used Chinese characters which take 3 bytes, then if the target elements are special symbols like expression symbols, the app may crash. Or if fails to download or display the picture, the data capture on that element may cause crash.

2. Performance: Go through main workflows and UIs to make sure the app still work with a good performance. When issues like lag, slow loading or even ANR occur, compare with the behavior of original app without the SDK to clarify if it’s caused by SDK or not.

Good data collect and transfer strategies is of great importance for the performance. The strategies define the content to collect, to what granularity (user visit, page visit, element click or more) the target data need to be, when to transfer the data, in what intervals, how many entries or what size of data are to be transferred in one time, what kind of data or what size of data are to be transferred on different network situations (Wi-Fi or different data plans), when to clean the data cached in the device, etc.

Here is an example that describes how the way and the content to collect affects the performance. Image a container control that shows scrolling images or videos. It may contain hundreds of hidden text or images. If go through and collect the data for each of these hidden elements, heavily lag or pause may happen when view the scrolling images or video. In this case, you may want to just collect the behavior data of the parent container.

Evaluating performance under different scenarios is useful to define efficient data collection and transfer strategies. Below scenarios can be taken into account:

• The data usage, time cost and other performance KPI when transfer one package on one device
• The performance KPI when transferring a big package on one device
• The performance KPI when transferring packages on multiple devices

For testers, one of the important check point is to validate the strategies are implemented correctly.

Accuracy

Speaking of accuracy, first of all, the behavior on all necessary controls should be captured. One fact is, it’s hard to capture most customized controls. So try avoiding or reducing the usage of customized controls. More attention should also be paid on the capture on gestures and controls like popup window or dropdown list.

Then validate the data is transferred and stored correctly in database. Think of below scenarios:

• Multiple click on the same control happens continuously. Each click should be counted.
• Data transfer from multiple devices at the same time are accepted correctly.
• No data transfer happens and no data is left in the device after the app is uninstalled.
• Transfer happens correctly when switching between different network scenarios.
• When exceptions happen, like the app exits, network shuts down, the device powers off, the data can be transferred correctly after resume.

Security

One of the most important check point is, make sure that the SDK uses https rather than http, so that other apps cannot access user data or call backend API of the app via the SDK integrated. Also, make sure sensitive data like user account, password and others are not captured and transferred.

If feasible, automation will help a lot on those repeated actions and stress testing. Tools like Monkey and Monkey Runner can also be leveraged.