Testing Strategy in Android — Part 2: The Pyramid and The Coverage
If you ever have heard about the Testing Pyramid, it’s probable that you might have associated or might have confused it with a Testing Strategy.
It’s usual to study about testing and then learn about the pyramid and take it as a rule, instead of a guidance tool.
In the first post of this series, i have explained what is in fact a Testing Strategy. You can read it here:
A Testing Pyramid is an ideal way to distribute the type of tests that you should have in your application, considering the execution and the maintainability, following this scheme:
70% Unit Testing — 20% Integration Testing — 10% UI Testing, resulting in 100%.
So, at an Android project we could apply the testing pyramid, where it would have a lot of unit testing (to validate classes or functions), a weighted number of integration tests(to validate the data flow through the units) and a small number of UI tests(because it might take more time to run).
However, an application Test Suite doesn’t necessarily need to fit this geometric form. Think about libraries like Room, Retrofit, Koin, Coroutines, RxJava… all of them are Android libraries, but none of them provides UI rendering capabilities.
It means that, the Pyramid, wouldn’t have a pyramid form actually, because the UI Tests that represent the top of it, couldn’t be applied.
It might(not necessarily) be like this figure on the left side, where only Unit and Integration Tests would be applied.
So, the pyramid represents an ideal testing suite distribution, based on a interconnection between the execution time and the maintainability on a front-end project.
It can’t be taken as an absolute rule because it doesn’t apply to all application types and also, the distribution can be done differently, depending on the testing strategy. Here’s an article with a different approach showing the Testing Trophy.
Here’s an example:
There are full applications or some parts of it that are simple “mirrors” of the backend API that returns the needed data to be exhibited at a screen. There aren’t business logics or data handling involved.
In this scenarios, what’s the best approach: repetitive unit tests of classes that act as simple proxies(a class calling the other one returning back values), or integrated tests that assert the return of a set of classes that work together at an application layer?
The answer is simple: it depends.
It depends on what is most important for the team(time, maintainability, execution costs, fidelity, etc…) at the moment, and also on how the project will scale over the time.
Through a product lifetime, things can change. The market, the requirements, the users, the tools, the frameworks and etc. The Testing Strategy should be evolved at the same pace, to fit what’s the best approach for the moment.
The only thing that is 100% sure is that a Testing Pyramid can’t tell you what’s the best approach, because it isn’t a Testing Strategy. It’s just a guidance tool.
It tells us what to test, but doesn’t tell us how to properly test it, application’s specificities aren’t considered on it, doesn’t spot the importance to focus on common failure points and doesn’t work as a documentation for the team. So, it isn’t a Testing Strategy.
Noah Sussman has reimagined the Testing Pyramid in a different way like it was a bug filter. It’s the same geometric form but turned upside down.
It gives more purpose to the testing pyramid, as it doesn’t look only like a geometric testing distribution propose. Noah’s Testing Pyramid makes easier to think about what’s the best test for a specific problem that might happen.
Having a purpose to test it’s better than have just a testing coverage percentage as a goal. It helps to create tests that actually adds value.
Testing coverage only express which lines of the production code are executed during the tests execution and nothing more. We will see this in the next example:
Here’s a class that validate names. There’s some rules in it, that will return a true or false if a name is valid or not.
And here’s it’s unit testings:
When a coverage tool, like jacoco, is executed to generate a report of the NameValidator class, it will be shown that 100% of this code is covered by tests. So, one may ask “ does it means quality?”. The answer is: not exactly.
Notice that this name validation isn’t completely done. The class NameValidator would return true, when the “isValid” function is called with the following strings, that doesn’t actually represent real names:
“123”, “John Doe#1984”, “Michael Jordan 🐐”.
Strings containing numbers, special characters or emojis would pass in the validations returning true, when it shouldn’t. The code to ensure that this cases are invalid weren’t implemented. Even tho, the test coverage report would say that it’s 100% covered.
That’s why Testing coverage doesn’t exactly express quality.
And that’s why using the Testing Pyramid(or any other geometric form) as a Bug Filter can helps us to put the focus on tests cases and possible failure points instead of testing coverage. It’s all about perspective.
In this post i’ve focused on the concepts about testing pyramid and testing coverage.
If you want to understand how to apply the Testing Pyramid in Android, you can check this series of posts written by Phellipe Silva.
Also, if you want to understand how to generate a Testing Coverage Report, you can check this post written by Rafael Toledo.
And here’s the complete list of references that i’ve read to write this post:
Thanks to Samanta Cicilia for inspirate me to write this, and thanks to Kellycroesy for improving my english!
Share the post with anyone who is learning about testing. Also leave a comment here, if you’ve ever had to apply the Testing Pyramid, or any other variations,to validate your application!
