How Many Tests Are Enough?

How Many Tests Are Enough?
How Many Tests Are Enough?

When trying to decide how to distribute our testing efforts and how many tests to write for each different type, it’s important to consider:

  1. Test coverage: How much of the code has a test that verifies its behavior
  2. Test execution time: How long does it take to run the test and get a result back
  3. Test feedback: How helpful is the feedback we get back from the test
  4. Test maintenance: How often will the test need to be updated

A unit test offers the least amount of coverage since it focuses on a single method. Many unit tests are needed to get high coverage. System tests or UAT cover a large area of the code and thus produce higher coverage

A unit test is very fast since it requires minimal setup focuses on a single method. Test execution is in the miliseconds. System tests or UAT cover a large area of the code and require setup steps and thus take longer to run, typically several seconds to minutes.

A unit test provides specific feedback as to what failed since the test is narrow in scope. So minimal troubleshooting is required in order to fix the code. System or UAT are wide in scope. A failed test informs us that there is a bug somewhere, but further troubleshooting is required to determine which integration point, or component, or unit needs fixing.

Since a unit test is narrow is scope, it only needs updating when the immediately code the test covers is being modified. Any other change will not impact the test. A system or UAT is wide in scope, and any change along the many parts that the test executes will cause a need for the tests to be updated.

In order to have maintainable tests that provide quick feedback and high coverage, we need to write many unit tests, followed by a smaller number of integration tests, follow by a smaller number of component tests and so forth.

This leads to the testing pyramid.

Most teams today use an inverted pyramid where most of the tests are manual User Acceptance Tests and System tests and are manually executed via the UI. Unit tests are non-existent or are too few to be beneficial.

Also check out the entire Agile Testing series: