Automated Testing – Executive Summary

Automated testing is something developers talk a lot about. But what is automated testing, and why is it important?

Unit testing usually means testing a piece of software in isolation, away from the rest of the code base. Sometimes this entire topic falls under the broad heading of unit testing. Unit testing typically doesn’t include integration testing, which is the testing of multiple pieces of software together.

At Don’t Panic Labs, these terms have particular meanings. We use unit testing to test an individual service, and we use integration testing to test many services together.

Unit Test Example

Below we are testing the Tax Rule Engine in isolation. We are using a Mock Tax Rule Accessor instead of a real Tax Rule Accessor.

With an integration test, we usually test the entire call chain and the resources used by the call chain. In the example below, we would test the real Order Manager, the real Tax Rule Engine, and the real Tax Accessor. The integration test could also hit a real database.

Unit and integration tests complement each other nicely. Unit tests allow us to have full control over service dependencies. Integration tests allow us to fully test the software as if it were running in production.

We have to be careful in letting it become a unit vs integration debate. We ideally want both. Both are valuable. The complement of unit and integration creates a better development pit of success.

Unit tests help in two ways:

• Improve quality. They are an important piece in the layered approach to quality we strive for.
• Improve design quality. If you can unit test it, it is often more loosely coupled.

Integration tests help in two ways:

• Improve quality. They are better at finding bugs than unit tests.
• Help reduce the stabilization phase in a project. Stabilization is that nasty phase of a project where a project never seems to end.

Both unit and integration tests help to improve quality, but neither is the best way to improve quality. Code reviews are the single best way to improve code quality. From the classic book Code Complete by Steve McConnell, code and design inspections find 55% – 60% of defects. In comparison, unit tests only find around 25 percent, and integration tests find around 45%.

Just because unit and integration tests only find smaller portions of bugs, they are an essential part of a layered approach to quality. No single technique will get you a super high code coverage number. That will only happen if we layer in many approaches.

Automated tests are not the be-all end-all for software development. Many engineers make automated tests the end goal of software. They are not. We want to create high-quality solutions that make customers happy. Automated tests help us deliver on that goal.