How Many Tests Are Enough?

  • Post author:
  • Reading time:3 mins read
How Many Tests Are Enough?
How Many Tests Are Enough?

When trying to decide how to distribute our testing efforts and how many tests to write for each different type, it’s important to consider:

  1. Test coverage: How much of the code has a test that verifies its behavior
  2. Test execution time: How long does it take to run the test and get a result back
  3. Test feedback: How helpful is the feedback we get back from the test
  4. Test maintenance: How often will the test need to be updated

A unit test offers the least amount of coverage since it focuses on a single method. Many unit tests are needed to get high coverage. System tests or End to End User Acceptance Tests cover a large area of the code and thus produce higher coverage.

A unit test is very fast since it requires minimal setup and focuses on a single method. Test execution is in milliseconds. System tests or UAT cover a large area of the code and require setup steps and dependencies and thus take longer to run, typically several seconds to minutes.

A unit test provides specific feedback as to what failed since the test is narrow in scope. So minimal troubleshooting is required to fix the code. System or UAT is wide in scope. A failed test informs us that there is a bug somewhere, but further troubleshooting is required to determine which integration point, component, or line needs fixing.

Since a unit test is narrow is scope, it only needs updating when the immediately code the test covers is being modified. Any other change will not impact the test. A system or UAT is wide in scope, and any change along the many parts that the test executes will cause a need for the tests to be updated.

So since unit tests are fast, provide immediate actionable feedback, and are easier to maintain, our testing strategy should consist of many unit tests, followed by a smaller number of integration tests, followed by a smaller number of component tests, and so forth. As the scope of the tests increases, so should the number of those tests. End-to-end tests have high code coverage and take longer to run, provide less actionable feedback, and are harder to maintain, so we should have fewer and fewer of them.

This leads to the testing pyramid.

Unfortunately, most teams today use an inverted pyramid where most of the tests are manual User Acceptance Tests and System tests that are manually executed via the UI. Unit tests are non-existent or are too few to be beneficial.

Also check out the entire Agile Testing series: