Ensuring Software Quality with a Layered Approach

The Beehive team (on which I work), as well as all the Nebraska Global companies, have made layered processes and techniques part of our day-to-day operations to ensure the quality of the software we are creating. It is very easy to make a statement like “by relying on more than one testing technique, we will be able to create more quality software,” and have it resonate with most folks. However, what I believe is more interesting is how we have gone about integrating a focus on quality through a mixture of techniques, processes, and culture.

Techniques

As explained by Steve McConnell in his book Code Complete, the average modal rate of defect detection for a single quality assurance technique is only 40%. This means we must use multiple techniques in order to create an acceptable level of quality. As we think about the different techniques used to find issues, it also becomes clear that various techniques are more effective at catching different kinds of issues. Our unit tests are most effective at validating individual pieces of functionality especially with accessing data and business-logic, whereas a QA person can look at how new functionality integrates and affects the overall workflow in a way that a segregated unit test cannot.

A study by Laurie Williams, Gunnar Kudrjavets, and Nachiappan Nagappan at North Carolina State University looked at the impact of adding automated unit testing to a team at Microsoft. It found that the addition of unit tests improved the quality of the code. The testers in this study still found bugs, but reported that they had to try harder to find bugs and – given the same amount of time – were “more effective at finding the more comprehensive defects that would have been discovered by customers rather than sticking to an isolated area of new functionality.”

In addition to unit tests and having an incredible QA staff, we also take advantage of the pull request functionality in GitHub to have our software architects or lead engineers complete code reviews prior to merge into our production branch. The primary goal is to review the architectural pieces and adherence to team coding standards.

Process

One of our processes that is especially helpful for our team is to rebuild our QA environment nightly. The automated steps to rebuild our QA environment are:

1. Back up every customer’s production database. It is probably worth noting that having a backup of data is something desired even if it was not needed to restore our QA environment.
2. Flip a bit on QA that stops any new data syncing from our QA test clients to the server for all customers tagged for rebuild.
3. Delete all QA databases for customers tagged for rebuild.
4. Restore the production database for each customer tagged for rebuild.
5. Update a few configurations including re-setting the version of the services used to match that in production.
6. Run each tagged QA customer through the release process that will take it from the current production version to the most recent release candidate from the production branch.

The release candidates are, at a minimum, generated daily from our master Git branch. So what this all means is that every morning we walk into the office and our QA environment is running exactly as if the release had been shipped to production. This gives us a high level of confidence that things will work the same when we release an update to our customers. It also helps ensure that we don’t see the disparity we did in the early days of the company where something that worked in our QA environment didn’t work in production. In Beehive, we have the luxury of being able to test with the same data that is in production. This is definitely not always feasible for teams, but there is immense value in having data as similar to production as possible.

Culture

Perhaps the most important element of integrating quality throughout the development process is having a culture that supports quality. Culture is one of those things that is a lot harder to measure than quantifiables such as the number of unit tests performed, percentage of code coverage, or number of defects caught prior to a release. At the end of the day, you can have as many techniques or processes as you want but unless the whole team has bought into the idea of taking responsibility for the quality of the software you will have a shoddy product. The primary reason that the Beehive team is able to have a single QA person for ~10 software engineers comes down to the fact that each of those engineers are doing their part to develop quality software. We still make our share of mistakes and have bugs that make it out the door, but there is also an underlying current throughout the team to get better at what we do and create a better product each and every day.

Conclusion

While our processes are far from perfect and we would never claim to have the silver bullet, having a layered approach to ensuring the quality of the software is something that everyone should strive for. It takes having a multi-faceted approach to testing through a variety of techniques, ensuring your processes support effective testing and confidence in the quality of your software, and a culture that emphasizes all team-members are responsible for the quality of the product.