3 Automated Testing Techniques, Part 3: Error Guessing

When writing tests for some new functionality, our goal should be to write the minimum number of tests with the highest likelihood of finding defects. To do this, we need a repeatable, consistent approach to defining our tests that will result in a similar outcome and quality of tests across the development team.

In the first part of this series, I covered basis testing. In part two, I continued with data-flow testing. In this third and final part, I will cover error guessing.

With this strategy, the developer looks at the code and uses intuition and experience to break the code using different inputs. Think of how many errors occur in software because of null data or bad data being passed into a method or a logic error that only occurs at a boundary condition of an “if” statement. This process involves looking at your program to identify any of the following:

• Boundary conditions
• Bad data inputs
• Good (but unusual) data inputs

Boundary conditions represent a particularly heinous type of logic error because they are often very hard to see in code or identify in testing unless you are explicitly testing for them. This is why this simple analysis can be so valuable. Boundary condition errors are most often “off-by-one” bugs, like in cases when you are using “>=” instead of “>” in a selection condition. They can also result from not properly checking for minimum and maximum values.

The image below shows the identification of a boundary condition in our method involving the MaxBaseSalary constant.

An analysis of the boundary conditions of this statement provides the following test scenarios:

• Salary < MaxBaseSalary o Covered by Test 1 if we modify to specify test value of MaxBaseSalary – 1 • Salary > MaxBaseSalary
  • Covered by Test 3 if we modify to specify test value of MaxBaseSalary + 1
• Salary == MaxBaseSalary
  • Not covered

Two of our boundary conditions are already covered if we change the Salary test condition of these scenarios to be a more specific value of MaxBaseSalary +/- 1. This demonstrates the ability for this layered analysis to minimize the number of test cases that need to be written for a program by modifying an existing test case scenario instead of creating an overlapping test.

Our analysis also shows that we did not have a test that covered the scenario where the employee’s salary was exactly MaxBaseSalary, so we need to add another test case to our matrix.

The testing techniques presented in this three-part series provide a strong strategy for testing your code. If you follow these three steps, your chances of success (finding issues) will go up. Will this guarantee you find all issues? No. Should I do this on all code? No. But this is a solid strategy that should be frequently leveraged.

For more information, I recommend looking at Code Complete by Steve McConnell.