It’s Just Config
Friday, 4:47 PM. An engineer opens a pull request on a billing service. One file changed. One line changed. The reviewer is already in their coat.
“LGTM — it’s just config.”
The PR merges at 4:58. By Monday morning, the team will have charged 3,800 customers two to four times each, refunded $2.3 million, and burned a quarter of their roadmap on the cleanup. The bug took eleven minutes to ship. It hid for forty-seven hours.
I made that team up. The story is invented. The pattern is not.
The four most expensive words in software are “it’s just config,” and I have heard them – and said them – more times than I want to admit.
The Fictional Bug
Let me walk through the made-up one first, because it is small and clean and it shows you the whole shape.
The team runs a billing service. Call it Helios. It charges about 40,000 customers a day through a third-party payment gateway. It has been quietly fine for years. One thing has been quietly true the whole time: when Helios calls the gateway, it does not send an idempotency key. The gateway has no way to tell a retry from a fresh charge. Nobody has noticed, because nothing has ever retried.
On a Friday afternoon, an engineer is chasing a sluggish dashboard. They trace it to slow charge calls, and they ship the obvious fix: drop the gateway timeout from 30 seconds to 3 and enable automatic retries. One configuration file, two values. The PR is essentially this:
payBridge:
- timeoutMs: 30000
- retries: 0
+ timeoutMs: 3000
+ retries: 3It is not a code change. There is no test to update. The reviewer skims it in under a minute and approves. CI is green by 4:55. It deploys at 4:58.
Saturday morning, the gateway slows down under weekend load. Charges start crossing the new 3-second cutoff. Helios records them as failures and retries. But the gateway processed each one. So Helios charges the cards a second time. And a third. The gateway, with no idempotency key, has no way to know it has already seen these requests.
The dashboards stay green the entire weekend. Helios is measuring “did our HTTP call return?” — and it didn’t, because it timed out. The fact that the charge actually succeeded on the other end is invisible to Helios’s metrics. The pipe looks healthy. The world is on fire.
Monday morning, the support queue tells the team what their monitoring did not. Forty-seven hours after a one-line config change. About $2.3 million in duplicate charges. About 3,800 customers who did not deserve the experience. About six hundred who will never come back.
That is what an unobserved, unreviewed, untested, unrolled-out config change looks like when the wind is wrong.
The Pattern in a Thousand Costumes
I picked a billing example because the dollar number is easy to feel. But the same pattern wears a thousand costumes:
A timeout. A retry count. A connection pool size. A rate limit. A feature flag. A WAF regex. A DNS TTL. A Kubernetes annotation. A queue depth. A cache expiration. A garbage-collector flag. An IAM policy. A log level. A connection-string suffix.
None of it feels like “real” code. None of it gets the treatment real code gets — the unit test, the careful review, the staged rollout, the rollback plan written before the merge. And so a change that would have died in code review, or been caught by a single test, or been canaried for half an hour before global rollout, instead walks straight into production untouched.
The thing that makes “it’s just config” so dangerous is not the change itself. The change is usually small. The thing that makes it dangerous is what the words turn off in the rest of the team. Reviewers skim faster. Tests do not get written. Rollouts are not staged. Alerts are not tuned. The whole defensive apparatus we built around code politely steps aside for the config change, because the config change does not look like code.
It is, of course, exactly like code. The values are not text. They are behavior. They change what the running system does, and they ride into production through the same pipeline, with the same blast radius, at the same speed. The distinction we make in our heads – “this is config, that is code” – does not exist in production. Production runs whatever you handed it.
The Receipts
I would let myself be talked out of this thesis, except that the receipts are unusually long, unusually well-documented, and unusually expensive.
Knight Capital, 2012. A deployment left an old, dormant code path live on one of eight servers. A feature flag – a config value – flipped a behavior nobody thought was reachable. Forty-five minutes of automated trading later, the firm had lost about $440 million and was effectively out of business. The post-mortem makes it clear: the change itself was small, the testing of the small change was light, the rollback path was unclear, and nobody was watching the right signal. Sound familiar?
CrowdStrike, July 2024. A “Rapid Response Content update” – note that they do not even call it a config change; it is content – slipped past a faulty validator and crashed roughly 8.5 million Windows machines globally inside a few hours. Airlines grounded. Hospitals offline. The change did not go through the same staged rollout as a sensor code release, because in CrowdStrike’s mental model, content updates were a different, lighter category. The world disagreed.
Cloudflare, July 2019. One WAF rule. A regex with catastrophic backtracking, pushed to every Cloudflare edge at once, with no canary. CPU on every server in the fleet went to 100 percent simultaneously. A large slice of the internet went dark for about 27 minutes. A regex is, mechanically, about as far from “real code” as you can get. It is also about as close to “executed by every server you own” as you can get. The two are not in tension. They are the same thing.
Every one of those teams had smart engineers, real review processes, and real test suites. What they did not have was the discipline to apply those processes to the things that did not look like code. And so the change went through the pipeline our culture designed for quick changes instead of the pipeline our engineering designed for risky changes — even though the change was, in fact, risky.
Config is Code
The fix is not heroics or a new framework. It is one simple commitment, applied with discipline, until it stops feeling weird.
Config is code.
The values are behavior. The pipeline that ships them is the same pipeline that ships your services. The blast radius is the same. Treat them the same.
What that looks like in practice is mundane and unglamorous, which is exactly what good engineering tends to look like.
- Review config changes like code. Same checklist, same questions, same depth. “Are these calls idempotent before we add retries?” is the question that would have saved Helios – and it is exactly the kind of question a real code review asks and a “LGTM, it’s just config” skim does not.
- Test config changes when you can. A contract test that fails when the configured timeout is shorter than the upstream’s documented worst case is not exotic. It is a half-hour of work and it would have killed the Helios bug in CI.
- Stage rollouts for anything global. Canary, blue-green, feature flag with a kill switch. If a change can affect every customer at once, it should not affect every customer at once. The CrowdStrike outage was not a unique act of nature; it was a unique decision not to canary.
- Alert on what the change actually affects. Not just CPU and memory. Charges per order. Login success rate. The thing the change was supposed to make better or might quietly make worse.
- Write the rollback before you merge. Three sentences in the PR description: how do we know it’s broken, how do we turn it off, who has the access. If you can’t answer those, you are not ready to merge.
The Reframe
Here is the part I want to leave you with. I am not asking anyone to slow down or to be more cautious or to ship less. The teams that ship the most are not the ones who treat every change like a NASA launch. They are the ones who treat every change with the appropriate amount of care for the blast radius it actually has.
The teams that get hurt are the ones whose care does not match the change’s blast radius. They review hard when the change looks scary and they skim when the change looks small. And the smallest-looking changes – the configs, the flags, the regexes, the rules – are precisely the ones whose looks deceive.
So the question I would ask after every “it’s just config” conversation is the one Helios’s reviewer did not ask: if this is wrong, what breaks, and how many customers feel it before we notice?
If the answer is “a lot, and probably tomorrow,” then it is not just config.
It never was.
