Quality Isn’t Tested In, It’s Layered In
We once delivered a project in six weeks. It felt fast. The demo landed, the client was happy, and we walked out of the room feeling like heroes.
Then we spent the next six weeks stabilizing it.
I think about that project a lot, because it taught me something that no estimate ever shows you: quality deferred is quality paid back, with interest. We didn’t save six weeks. We borrowed them. And the interest rate on that loan turned out to be brutal.
I have talked to a lot of teams over the years, and almost every one of them treats quality as a phase. Build the thing, then “do quality” at the end. Write the code, then test it. Ship it, then watch it. The problem with that model is that by the time you get to the end, the cheapest moment to fix anything is already behind you.
Quality Has More than One Face
When most people say “quality,” they mean does it work. That is real, but it is only one layer.
There is functional quality — does the software do what it is supposed to do? There is structural quality — is the code clean, secure, maintainable, the kind of thing a future developer can change without holding their breath? There is process quality — are we building it the right way? There is experience quality — does it feel right to the person using it? And there is operational quality — does it actually hold up in production at 2 a.m.?
Each one rests on the one below it. Skip a layer and the whole stack wobbles. A beautiful UI on top of a flaky data layer is just a nicer way to be wrong.
But here is what I have learned: naming the faces of quality is not the hard part. The hard part is catching problems before they get expensive. And the cost curve here is not gentle.
A defect caught while you are still arguing about requirements costs about 1x to fix. Catch it in design, maybe 5x. In code, 10x. In testing, 20x. In production? The old IBM number is 100x or more, and honestly that feels low once you add the support calls, the trust you lose, and the roadmap you set on fire to deal with it.
So the goal is not “test more.” The goal is to give every defect as many chances as possible to die young.
That is what layering means.
Five Layers, Because No Single Net Catches Everything
I think of quality as five layers, each one a different net hung at a different point in the process.
- Prevention stops bugs before they exist. Clear requirements, a quick design review, a threat model, strict types, a linter. The cheapest bug is the one you never write.
- Detection finds bugs as the code is written. Code review, static analysis, dependency scanning, a second set of eyes. This is where a reviewer notices that your new endpoint logs the full request body, password and all.
- Verification proves the code does what you wrote it to do. Unit tests, property-based tests, contract tests. This is where a test pins down your currency rounding so a regression six months from now fails the build instead of the customer.
- Validation proves the whole thing works, and that you built the right thing. End-to-end tests, load tests, acceptance testing. Verification asks “did we build it right?” Validation asks “did we build the right thing?” Those are different questions, and you need both.
- Observation is the part everyone forgets, because it lives after the deploy. Logging, tracing, alerting, real user monitoring. Quality does not stop when the code ships. Half the time, production is where you learn what your tests forgot.
No single one of these is enough. Unit tests miss integration bugs. Integration tests miss UX problems. Code review misses runtime behavior. Monitoring misses logic errors. Layer them and a defect has to slip past all five to reach a customer. Miss one layer and you are betting everything on the others holding.
Let Me Make This Concrete
Imagine a billing service. Call it Helios. It charges about 40,000 customers a day through a third-party payment gateway, and it has worked fine for years. Quietly, since day one, it has called that gateway without an idempotency key. Nobody noticed, because nothing ever retried.
One Friday afternoon, an engineer is chasing a sluggish dashboard. They trace it to slow charge calls and ship a quick win: drop the gateway timeout from 30 seconds to 3 and turn on automatic retries. One line of config. A teammate already in their code reviews it in about 19 seconds — “LGTM, it’s just config” — and it deploys at 4:58.
Here is what nobody is thinking about. A 3-second timeout is shorter than the gateway’s own worst-case processing time. A timeout on our side does not mean the charge failed on their side. And without an idempotency key, the gateway cannot tell a retry from a brand-new charge.
Each of those is harmless on its own. Together they are a loaded gun.
Saturday morning, the gateway slows down under weekend load. Charges start crossing the 3-second line. Helios records them as failures and retries them, but they had already succeeded. So it charges the cards again. Some two times, some four. Over the weekend, 14,000 duplicate charges go out, $2.3 million wrongly billed to thousands of customers.
And the dashboards stay green the whole time, because Helios counts a timeout as a “failure,” not as a double charge. The bug shipped in eleven minutes. It hid for forty-seven hours. We found it Monday, from the support queue.
I made Helios up. I did not make up the pattern. Knight Capital lost $440 million in 45 minutes in 2012 because a deploy left stale code running on one of eight servers. CrowdStrike took down millions of Windows machines in 2024 with a content configuration update that slipped past a buggy validator. Cloudflare knocked itself offline globally in 2019 with a single regex rule pushed everywhere at once. Retry-without-idempotency is such a well-known foot-gun that Stripe and AWS built their public APIs specifically to prevent it.
The thing that gets me about the Helios story is not that someone made a mistake. People make mistakes. The thing that gets me is that the mistake had a clear, free path all the way to production. It skipped every layer at once.
A “config is code” rule would have made it get reviewed like code. A reviewer with a checklist would have asked, “are these calls idempotent before we add retries?” A single test simulating a slow gateway would have turned it into a red build. A load test with injected latency would have surfaced it in staging. An alert on charges-per-order would have caught it in minutes instead of two days. Any one of those stops it. Helios had zero.
That is the whole argument for layers in one story. You are not trying to be perfect at any single layer. You are trying to make sure a single miss is not a catastrophe.
What to Actually Add on Monday
I am not telling you to go build all five layers this quarter. That is how good intentions die. Pick your weakest layer and add one thing. Here are the ones I reach for first.
- Prevention: Treat config, flags, and infrastructure as code — same review, same tests. Most “it’s just config” incidents start with those four words.
- Detection: A real PR checklist (security, error handling, idempotency, rollback) plus automated secret and dependency scanning. Make the safe path the default path.
- Verification: A regression test for every production bug you find. No fix merges without a failing test first. Your test suite becomes a memory of every way you have been burned.
- Validation: A staged rollout — canary or feature flags — with a rollback you can hit in one click. You will ship a bad change eventually. Decide now how fast you can take it back.
- Observation: One business-metric alert, not just infrastructure metrics. CPU and memory looked fine all weekend at Helios. “Charges per order = 2” did not.
Pick one. Start this week. Direction beats precision here. You do not need a dashboard team and a six-month initiative. You need one more net.
Further Reading
- CISQ, The Cost of Poor Software Quality in the US, 2022
- IBM Systems Sciences Institute on the relative cost of fixing defects
- SEC order on Knight Capital (2013)
- CrowdStrike’s post-incident review (2024)
- Cloudflare’s July 2, 2019 outage writeup
- Stripe’s idempotent requests
- AWS Builders’ Library on timeouts, retries, and backoff
