Android Screenshot testing for UI

(Automated testing that helps prevent visual regressions in Android apps)

Testing on Android nowadays tools like Mockito/Mockk and Espresso make it easy to write unit and UI tests for Android. At Runtastic, our Android engineers share a lot of UI components across our apps. With screenshot testing, we can test our UI components against visual regressions under various conditions. In this blog post, we’ll show you how that works and how you can apply screenshot testing for your projects too.

Probably every mobile developer knows these small but annoying bugs: The general toolbar of the app is restyled, and it looks good on 29 out of the total 30 screens but on one of the screens some UI elements moved unnoticed onto the wrong positions. Something like this is easily overseen in manual testing since the QA would need to test the change on every device.

Wouldn’t it be cool to have a convenient and easy way to find such regressions quickly and keep track of UI changes? Snapshot testing solves this by automatically comparing the previous and the new build.

What is Android Screenshot testing?

Android Screenshot testing is the process of taking screenshots on Android devices while testing them. This is primarily done to check how the UI looks on different devices.

This makes it difficult for the developers and testers to develop and deliver Android applications that provide a consistent user experience over such a large variety of devices. Even a small discrepancy in an application feature can lead the users to uninstall the app and hence a great loss.

Let’s take a look at some side effects of Android fragmentation:

• Difficulty in testing apps: Due to the availability of a diverse range of Android devices, it becomes really difficult to test the apps. It is not possible to test all the features of an application on every single Android version available.
• New apps do not run on older Android versions: There are many new applications available in the Play Store that do not run on the older versions of Android. For example, you can not use the Uber app, on Android devices below 4.0.3.
• Security risk: Many Android users are not even close to the latest upgrades and some of them will never even get to use them at all. Applications that are infected with malware find it easier to enter phones that are not running the latest upgrades.

Android Screenshot testing can help in providing a consistent user experience at multiple devices. Screenshot testing of Android is basically a way of taking a screenshot of an app and then comparing it to another screenshot taken while the test is running.

But performing Android screenshot testing manually can be a tedious task because the operating system versions, hardware specifications, screen sizes, and system UI all differ from device to device and brand to brand, test results will vary in the same way.

To know more about it, Click here.

HOW DOES SCREENSHOT TESTING WORK IN GENERAL?

Screenshot testing uses instrumented UI tests to render a screen or a portion of a screen to an image. This image is then compared to a “reference image”. If both images are equal, the test succeeds — otherwise it fails.

A screenshot test is quite similar to a UI test. The only difference is that for assertion the rendered image is compared to a reference image.

Screenshot Testing Libraries

Android Firebase Test Lab Api provide the screenshot ui testing.

There are so many third party library also available for Screenshot Testing are :

https://github.com/facebook/screenshot-tests-for-android
https://github.com/cashapp/paparazzi
https://github.com/pedrovgs/Shot

1. Screenshot Test for Android

screenshot-tests-for-android is a library that can generate fast deterministic screenshots while running instrumentation tests on Android.

We mimic Android’s measure(), layout() and draw() to generate screenshots on the test thread. By not having to do the rendering on a separate thread we have control over animations and handler callbacks which makes the screenshots extremely deterministic and reliable for catching regressions in continuous integration.

We also provide utilities for using screenshot tests during the development process. With these scripts you can iterate on a view or layout and quickly see how the view renders in a real Android environment, without having to build the whole app. You can also render the view in multiple configurations at one go. Check here to integrate it.

2. Paparazzi

The screenshot test library we are using is Paparazzi so we can ditch the whole emulator topic and all the problems that the solution would bring. Instead, the tests are fast to execute, cheap to verify on the CI, and easy to set up.

Under the hood, Paparazzi is using the layout library from Android Studio which also renders the preview for your XMLs. But instead of rendering the XML inside Android Studio, Paparazzi generates a PNG out of it.

3. Shot

Shot is a Gradle plugin and a core android library thought to run screenshot tests for Android. This project provides a handy interface named ScreenshotTest and a ready to use ShotTestRunner.

How to Integrate Firebase Test Lab library

Firebase Test Lab includes a library (testlab-instr-lib) that you can use to process any screenshots you take with AndroidX’s ScreenCapture when running instrumentation tests, such as tests written using the Espresso test framework. This section describes how to create ScreenCapture objects with the AndroidX library and how to process them using testlab-instr-lib.

After your instrumentation test has run, you can view the captured screenshots in the Firebase console.

You can download the sample app from here.

Step 1. Add the screenshot library to your project

1. In your test project’s root-level (project-level) Gradle file (build.gradle), add Google's Maven repository to every repositories section:

buildscript {

  repositories {
    // Add the following line:
    google()  // Google's Maven repository
  }

  dependencies {
    // ...

    // Check that you have the following line (if not, add it):
    classpath 'com.google.gms:google-services:4.3.8'  // Google Services plugin
  }
}

allprojects {
  // ...

  repositories {
    // Add the following line:
    google()  // Google's Maven repository
    // ...
  }
}

2. In your module (app-level) Gradle file (usually app/build.gradle), add a dependency for the Test Lab screenshot library.

dependencies {
  // ...
  // Add Test Lab's instrumentation test screenshot library:
  androidTestImplementation 'com.google.firebase:testlab-instr-lib:0.2'
  // ...
}

3. In your test’s AndroidManifest.xml file, register the FirebaseScreenCaptureProcessor in a meta-data tag within the <instrumentation> element. You can also specify the processor as an argument in AndroidJUnitRunner instead (see the AndroidJUnitRunner reference documentation for instructions on how).

<instrumentation
  // Check that you have the following line (if not, add it):
  android:name="androidx.test.runner.AndroidJUnitRunner" // Specifies AndroidJUnitRunner as the test runner
  android:targetPackage="com.your.package.name">

// Add the following:
<meta-data
  android:name="screenCaptureProcessors"
  android:value="com.google.firebase.testlab.screenshot.FirebaseScreenCaptureProcessor" />
</instrumentation>
...

4. In your app’s AndroidManifest.xml file, add the following lines within the <manifest> element:

 <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

5. In your AndroidManifest.xml file, specify system permissions for your app by adding the following lines within the <manifest> tag. If you're testing on Android 10 (API level 29) or higher, omit the WRITE_EXTERNAL_STORAGE permission (your app does not require this permission in order to read and write screenshots to the device).

<manifest ... >
    <!-- WRITE_EXTERNAL_STORAGE is not needed on Android 10 (API level 29) or higher. -->
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.INTERNET"/>
    ...
</manifest>

Step 2. Take screenshots during your test

At any point in your test where you want to take a screenshot, call the Screenshot.capture() method from the AndroidX library. This produces a ScreenCapture object. When you call process() on the ScreenCapture object, it gets processed using the ScreenCaptureProcessor that's registered in your AndroidManifest.xml. Note that the BasicScreenCaptureProcessor is used if no processors are registered. Since you registered the FirebaseScreenCaptureProcessor, your screenshots will be processed via FirebaseScreenCaptureProcessor and will be available for you with your results when you run your test with Firebase Test Lab.

Example use cases for creating a ScreenCapture:

1. Take a full ScreenCapture on a API Build.VERSION_CODES.JELLY_BEAN_MR2 and above:

Screenshot.capture()

2. Take a ScreenCapture of the Activity on any API level. Note this is the only option for devices that are below Build.VERSION_CODES.JELLY_BEAN_MR2.

@Rule
  public ActivityTestRule<MainActivity> activityRule = new ActivityTestRule<>(MainActivity.class);
...
Screenshot.capture(activityRule.getActivity());
...

Example use cases for processing a ScreenCapture

1. Process a ScreenCapture via the FirebaseScreenCaptureProcessor:

Screenshot.capture().process();

2. Process a ScreenCapture via a specified ScreenCaptureProcessor (this allows you to skip registering the processor):

Set<ScreenCaptureProcessor> processors = new HashSet<>();
processors.add(new FirebaseScreenCaptureProcessor());
Screenshot.capture().process(processors);

3. Set the name and format of the ScreenCapture and process it using the registered processor:

Screenshot.capture().setName("myscreenshot").setFormat(CompressFormat.JPEG).process();

Step 3. Build and run your test

1. Build your app and test APKs (see Test your app for instructions).
2. Upload the APK files to the Test Lab dashboard of the Firebase console.
3. Finally, run your test.

Step 4. View your test screenshots

After your test has completed, you can view any screenshots taken in the Firebase console.

1. In the Tests tab, select your completed test, then click the Results tab.
2. Select your test again, then click the Screenshots tab that appears.

Advantages and disadvantages

Pros

1. Less costly to write and maintain than UI tests. Mostly involves inflating the view, setting its state and taking the snapshot. It does require little or no interaction with the UI in most cases. If needed, it is possible to use Espresso or UiAutomator with snapshot testing, they are not mutually exclusive.
2. Much faster to run than UI tests. Depending on its complexity each test can run and record even in less than a second. Snapshot verification is a bit more time-demanding because it also involves comparing the snapshots with their corresponding references. I will also devote one chapter to reducing snapshot test execution times, for those having a large bunch of them.
3. Catches visual bugs that only snapshot tests can. UiAutomator/Espresso tests can verify whether a view is shown, but snapshot tests are much more precise at asserting whether the view displays as expected.
4. Help improve communication with designers and/or translators. Snapshot tests require teammates to confirm that the snapshot reflects the expected state of the view. By including designers and translators into their approval, we can catch design and translation bugs early before merging. The outcome is a decrease of the development costs of a feature, as well as an increase in the quality of the app overall.
5. Very scalable. Write one test and make it run with every possible configuration. This is straightforward with Parameterized tests, what I will explain in the next article. However, if not done carefully, it can exponentially increase your building times. I will also give some advice on writing good snapshot tests in the following post.

Cons

1. Any change in a view is treated as a regression bug unless a new test is recorded. You’ll often find yourself in a situation in which you modify some layouts due to new requirements, push your code, and it fails on the CI because you have not recorded a new Snapshot of that layout reflecting those changes.
2. Still not 100% automated. requires manual verification to know what is correctly displayed and what is not. Take the previous image as an example: How can you decide that the layout is broken without a human evaluating it? Or that a translation in a language is right? Or the colours in Dark mode are the ones expected?
3. Flakiness. When getting into more complex views, flakiness starts showing up for apparently “no reason”. But, take a breath. Most of these flakiness occurs for a reason and can be avoided. I will devote a full blog post on explaining how to find and get rid of such sources of flakiness.
4. Emulator configuration. The most common approach to snapshot testing is to use emulators because they are the “cheap” option. However, this is more cumbersome than it looks like. You’ll find yourself recording snapshot tests on the wrong device/emulator or wiping data from the emulator quite often to avoid errors due to insufficient storage. Verifying snapshot tests on the CI emulators? watch out for freezing emulators. And the show goes on… But again, do not worry about this too much. I will give some tips to mitigate all these issues in another post of this series.

Thanks for reading…

Screenshot Testing For Android >> How It Works

Snapshot Testing in Kotlin and Android

An introduction to snapshot testing on Android in 2021 📸