Why We Test
Everyone knows why developers test and we have been talking about it in previous posts. In case you need a reminder, here it is: software testing is performed to verify that the completed software package functions according to the expectations defined by the requirements/specifications. The overall objective is not to find every software bug that exists, but to uncover situations that could negatively impact the customer, usability and/or maintainability.
A Technical Definition of Testing
So basically, a test is a unit of work that controls the inputs of another program, called the SUT (System under Test). It then verifies that the outputs meet a certain criteria. A test output is used to determine whether the test passed the verifications or failed.
What Makes One Test Different From Another?
Tests types vary based on different dimensions including:
- the goal of the test
- the System Under Test (SUT)
- who is responsible for designing and writing the tests
- who is running them
- & more
Surprisingly, there is no standard or a widely-accepted definition of test types. Even our good friends from Google have faced this non-standard language issue and wrote about it in this article. They define “tests” as small, medium and large.
The problem with those terms is that they’re far too generic to be useful. Different people might say different things about test types.
Making Sense works on different kinds of projects which use different technologies. As a product and service software company, we have adopted common language and concepts around this topic.
The Test Types We Use
We found that the following three test type definitions work very well for us. They’re specific, concise, simple and clear. They also help us to communicate efficiently when it comes to testing in our agile development process:
- Unit tests: Test a component in isolation with mocked dependencies
- Integration tests: Test a component in integration with some external dependency
- End-to-End Test: Test the entire application as a whole
Note that the most accepted test type definitions are the ones for unit and end-to-end tests. When it comes to talking about integration tests, there’s a lot of confusion and diverse opinions about what they comprise. We’ll try and clear that up right now.
What Works for Us
Clear Definitions of all 3 Test Types.
If you take a look at Table 1, we describe the vision we have here at Making Sense regarding automated test types. It has helped us to build a common domain knowledge among our team members. As a result, we feel that we are able to communicate much more effectively in our goal to support our agile methodology.
The Pros & Cons of Each Test Type
Naturally, each test type comes with its own pros and cons:
- Help developers in the dev process
- Easier to get high code coverage
- Cheap to write due to isolated, controlled environment
- Fast to run due to small SUT and mocked deps
- Easy to add to a CI system
- Easy to automate (due to unit test tools)
- Might be hard to maintain if relying excessively upon mocks, and internal, non-stable interfaces and data structures
- Stable and easy to maintain because they typically use the SUT public interface
- Able to catch bugs that are difficult or impossible to detect in unit tests
- Contains interface defects in the integration phase before the system is deployed to the target environment
- Test backward compatibility between APIs
- Sometimes it might be difficult to set up the running environment with actual dependencies (docker comes to the rescue in most instances)
- Tests the whole SUT in a real-world environment
- Emulates the end-user experience before deploying to the final environment
- Prevents defects from being transferred to the final end-user environment
- Requires a mirror of the production environment
Regardless of the terminology you adopt, it’s a good idea to implement a 3-level approach like the one described above. Of course you’ll need heavy doses of common sense so that you end up making sensible decisions. But we are confident that with this approach, you’ve got the tools you need to contain the most number of bugs with the least amount of time spent during the development phase of your next project. Hope this helps!