Android Instrumented Tests: testing types and how they work
I’ve been developing digital products for a while and I’ve been in situations where developing automated testing was considered a separate step, instead of something that is part of the development process. It happens pretty often.
I bet you can relate to this situation: a team rushing with the development of a new feature because they were requested to not implement automated tests so they can release it faster. An then it is said that “let’s release it now and then implement tests later when we have the time to prioritize it”.
Usually, we implement automated tests to prevent the users to deal with bugs in production. So, what’s the point to release something without ensuring the quality we should deliver to them?! 🤷♂️
It’s a very odd situation, right? Quality is essential, during the application development process, so the users have an app that meets their needs in their devices, with fewer or no unexpected behaviors that might impact the user experience.
As developers who care about what is released to the user, we must be able to explain to our team/company why automated tests are very important and why we can’t take quality for granted.
Tests help to ensure the quality of an application and are the foundation of everything that is developed by the team.
Testing Types
When it comes to Tests, there’s a variety of different types of them.
These are the most known among them: Unit, Integration, Functional, Manual.
- Unit Tests will ensure that the small units(a set of 1..n classes/functions) of an application works as expected in isolation validating the inputs and outputs of them.
- Integration Tests will ensure that the units of an application work as expected when put together, interacting between them.
- Functional Tests will ensure that an application works as expected, by validating an end-to-end UI navigation through the features according to the user requirements, to simulate what a user would do.
- Manual Tests have the same goal as a Functional Test, except on how they will be executed. As it’s not going to be executed by a machine, it can be prone to human errors and it’s also slower than an automated test.
Instrumented Tests can relate to some of them, depending on what is your testing strategy.
Instrumentation
Before we jump into Instrumented Tests, it’s important to learn more about what is Instrumentation:
To instrumentate something in the context of software development is about the usage/creation of tools that provide ways to manage, measure, or control a system. Examples: Analytics, Crashlytics, Logs, Android Instrumentation, etc.
When it comes to Instrumented Tests, the Android framework allows us to do it, by driving the lifecycle of the app to be tested and also by doing interactions with the UI and assertions on the current state.
Instrumented Tests
An Instrumented Test is executed on a Device or Emulator and with the help of the Android Instrumentation, it’s possible to get information about the app, in a way that is possible to drive it while testing.
There’re two types of Instrumented Tests:
UI Instrumented Test
This is the most widespread of them. It’s a UI Test focused on doing assertions that validate the state of the screen under test, according to the inputs that a user would do, when interacting with it. These validations and interactions can be done with Espresso.
Here’s a simple instrumented test code example:
It can be executed on a real device or an emulator.
By starting the Instrumented Test’s Gradle task(connectedAndroidTest), right after build, two apk are generated, and then, both of them are installed in the device. One of them is the apk of the built App, the other one is an Apk that has the Test code inside of it, which will be used by the Android Instrumentation process, to interact with the App.
It can be illustrated by the following figure:
It’s important to notice that Espresso provides a very powerful API, that can be used to validate just one screen(Activity, Fragment, View) in isolation, per test. This means that is not necessary to validate more than one screen at a time.
There’s no need to do navigations through screens while testing because it would take more time to execute the test and also, it would make it more complex to set up dependencies, data, and so on.
With Espresso, if well configured, you can open any activity or fragment in isolation and validate only them.
As you can see, it can be like a “Unit test” where the unit would be a screen. In this scenario, it means that the presentation layer of your application is the only one that would be executed while testing. So, the others layers(domain layer and/or data layer) could be mocked. It saves a lot of time when it comes to execution. Try to isolate your Fragment/Activity as much as possible injecting mocks/fakes instead of the real implementation of its dependencies.
It can also be considered an “Integration Test”, if the team strategy for testing is to execute the Screen and let it communicate with other layers of the application architecture, allowing it to do network requests(to a MockWebServer) or any other data handling.
Also, it’s possible to do end-to-end UI validations, navigating through screens, with Espresso, but usually, it would become more like a Functional Test, and sometimes it’s better to use Appium because it has ways to provide a high-level validation(Gherkin + Cucumber) API that has better abstraction to be tied to Functional requirements than an Instrumented Test. But, consider that Appium is slower than Espresso. So, choose what’s best for your Testing Strategy.
Unit Instrumented Test
There are some regular classes, that aren’t related to UI rendering (Fragments/Activities/Views) which still depends on the Android Framework and should be validated with Unit Testing.
In that case, this unit test needs to be executed in an environment that has all regular Android framework classes loaded. This environment won’t be the JVM, it should be the ART(Android Runtime). To be able to do it, a device/emulator will host the execution of this unit instrumented test. Such tests can take advantage of Espresso, JUnit, and any mocking library that applies.
To be more concrete and exemplify the concept, imagine a class called FileUtils responsible to handle all related file operations in an application. In this class, there’s a function called getInternalDirectoryFilesListSize which returns the number of files inside a specific directory, in the app's internal storage.
It’s noticeable that this class doesn’t have anything related to Activities, Fragments, or Views. It’s a simple class, that depends on a Contextparameter and interacts with File class, both Android framework’s classes. The context is needed to set the app’s directory(internal storage) which will be searched and have its size returned.
So, it won’t be possible to implement a regular Unit Test, as the JVM doesn’t have Android Dependencies loaded on its environment (The Roboletric framework might come through your mind. I’ll talk about it in a bit..). The Context is created on the android runtime, as it’s tied to an Application, Activity, or Fragment.
That’s the case where it’s possible to implement a Unit Instrumented Test, which will ensure that the function works and returns the right information about the directory, created at the app's internal storage.
Notice: Unit Instrumented Tests should be in the src/androidTest sourceSet, not in the regular Unit Tests sourceSet src/test. Otherwise it wouldn’t run on a device.
The above test creates 3 files on a specific directory and calls FileUtils to get the number of files created on this directory and then assert if it is correct as expected.
A Unit Instrumented Test is very alike a regular UI Instrumented Test. It is executed on a device or emulator, and also generates two different apks(Test apk and App apk). The instrumentation is started, in the same way, on a separated thread, but the difference is that nothing related to UI is started unless the test code does something related to it.
Roboletric
You might have heard about Roboletric, a famous android testing library, that makes it possible to implement tests and run on JVM(your local machine), over classes that have Android dependencies, without the need to run the test on devices/emulators. So, it’s also possible to implement headless UI Tests with Roboletric.
The library loads a fake/mock instance of almost all the Android framework classes on the JVM, and they call it shadows. This process might bring an overhead, in comparison to a regular unit test of a pure java/kotlin class without android dependencies.
You can read more about the library here.
So, it’s important to notice that, if a test is implemented and executed with the help of Roboletric, this test might lose fidelity in comparison with an (Instrumented UI/Unit) test executed on a regular device. On the other hand, it's faster.
Then it’s up to you and your team, to decide which approach will be better for your app Test Strategy.
That’s it, I hope this article might’ve helped you to understand more about Instrumented tests. If you have any thoughts about it, please leave a comment below. Also, feel free to reach me on Twitter: @natanximenesdev
Thanks!
References:
You can learn more about Test Strategy in these two following articles:
And here’s the complete list of articles, videos, and books that I’ve used as references to write this post:
