Not all UI tests are the same

Introduction

Typically, when we think of UI testing, we think of simulating user behavior, such as when a user enters their email and password and clicks the login button. However, modern UI testing can do much more than just emulate user behavior.

In this article, we will explore the following topics:

• Different types of UI tests
• Understanding what different UI tests CAN and CANNOT verify
• What to check with each type of UI tests
• How to start with UI testing

All examples will be based on the "Mood Tracker" application. Let's explore 3 screens, as all our test cases will be based on these screens.

Different types of UI tests

End-To-End tests ensure that the application works with the production environment. This means that all application layers work correctly.

UI tests with fake data verify different states of the screen (no data, successfully loaded data, network is unavailable, and much more).

Pixel perfectness tests ensure that a component or screen looks correct in different states. Here, we usually use screenshot testing.

Accessibility tests check if everyone can use a user interface, including people with disabilities.

End-To-End tests

End-to-end tests verify the interaction between the UI, back-end, database, and other application layers.

Let's explore the main characteristics of such tests:

• Entry point: This is similar to the entry point of the application (we often start such tests from the splash or home screen).
• Navigation: They usually navigate through several screens starting from the entry point.
• Data: We use the production or staging environment (staging environment: replica or copy of the production environment) to ensure interaction verification with the production environment.

UI tests with fake data

UI tests with fake data verify the different states of the components or screens of the application and use the fake data to simulate all cases.

Main characteristics:

• Entry point: Required screen (if the UI testing framework allows us to do it)
• Navigation: Depends on the goal of the test case (they use fake data):
  • Screen tests (Screen component verification): No navigation needed
  • Navigation tests (Navigation verification): Navigation needed
• Data: Fake data

This screen displays the time and we can verify that the time is displayed in the correct time format without updating the values in the "Settings" screen. So, let's update our scenarios.

Pixel perfectness tests

End-To-End and UI tests with fake data can find many issues in the app. However, they cannot find pixel perfectness issues because they verify that data is displayed on the screen, but they don't check that data is displayed in the right place, that views don't overlap each other, that text doesn't cut off, that UI elements use the right color, font size, and much more.

If we look at this image, we can find the difference even if both screens have similar data.

So, pixel perfectness tests verify how content is rendered on the screen. We test components and screens with static data using this type of tests. When the UI is changed, the screenshots should be updated (usually it's done with a single command).

So, let's choose a few basic test scenarios which can be covered for the home screen with screenshot tests:

Often you can find a situation where a UI component or screen works with the default font size, but when the font size is large, you can see some problems in the UI. In this example, when the font size is large, some of the day abbreviations contain 2 symbols instead of 3 (e.g. "Mo" instead of "Mon").

I highly recommend checking screens with different font sizes and scales. I saw such issues in many apps.

Accessibility tests

The application may function well and have a good design, but that doesn't make it accessible for everyone. Often accessibility has a low priority and people with disabilities do not have the best experience when using various applications. Accessibility tests verify that our application is accessible to everyone.

Here are the common issues that can be verified with accessibility tests:

• Incorrect screen structure
• Missing labels for UI components
• Incorrect labels for complex components
• Small clickable elements, such as buttons and icons

Android recommendations:

• Before creating UI tests, I recommend verifying your application with the "Accessibility Scanner" app.
• Recently, the "Smart UI Check Mode" feature was added to the Android Studio Iguana, and you can verify if your composable functions are accessible without running the application.

I recommend covering all custom components (components from the design system) with accessibility tests.

How to start UI testing?

As you can see, the different types of tests have their advantages and disadvantages. If your application already has had UI tests, you will see what needs to be improved or which types of tests you can introduce to your project. However, if you are new to UI testing, you may have the question, "What is a good way to introduce UI testing to the project?".

Here are a few recommendations:

• First, we should prioritize E2E testing for the most critical use cases to ensure that the features work correctly and interact smoothly with the production/staging environment. Keep in mind that E2E testing tends to be slow and flaky. So it's a good place to start, but I recommend focusing more on other types of tests.
• Next, you can start adding UI tests with fake data and Pixel Perfectness tests, as they both check components/screens in isolation, and you can check all states and errors for your screens.
• To find the important screens to test, I suggest looking at the analytics. It will show you which screens and scenarios are used more than others in your application.

Conclusion

There are several types of UI tests:

• End-To-End tests ensure that your application works properly with the production/staging environment, but they are slower and less reliable than other types of test.
• UI tests with fake data check various states of your screens and use fake data; this makes these tests fast, but you cannot be sure that your application works well with the real environment using only this type of test.
• Pixel Perfectness tests verify that all your components and screens are rendered properly and placed correctly on the screen. You can also check your screen with different configurations like color modes, font sizes, display scales, layout directions, etc.
• Accessibility tests checks that everyone can use your application using gestures, voice, or special devices.

As you can see, if we want to make sure that your application works correctly, we need to check many scenarios:

• Verify that your application has no problems interacting with the production environment
• Check all states and errors for components and screens
• Verify that everything is correctly rendered and positioned on the screen
• Make sure that your application is accessible to everyone

This requires us to combine different types of UI testing.

I would like to thank Alexey Alter-Pesotskiy, Alexey Bykov, Ekaterina Basova, Evgenii Matsiuk, Jovche Mitrejchevski, and Sergio Sastre Florez for reviewing the article.