Making Operational Decisions Explainable

Many production incidents begin long before anything crashes. They begin when software quietly makes the wrong operational decision. It stops the wrong node, skips the backup that mattered, or cleans up data it should have kept, and it does so confidently, with no error and no alarm. The action executed perfectly. The decision was wrong.

That, I’ve come to think, is the harder problem in operational software. Executing an action is usually the easy part. The difficult part is making the decision understandable enough that you can trust it under pressure: during review, at 3 a.m., or six months later when someone else inherits the code.

When software makes these calls without a human in the loop, every action carries a policy behind it: a controller that deletes a snapshot has decided, by some rule, that deletion was the right move. When that policy is invisible, operating the system means reverse-engineering its reasoning from the outside, usually at the worst possible time.

I ran into this while building a Kubernetes operator that snapshots Ethereum staking nodes. Taking a consistent snapshot of a running stateful node takes several coordinated steps: stop, snapshot, and restart. It only helps on a healthy node, since snapshotting a broken one just preserves the fault. It also has to respect a maintenance mode, a retention limit, and a restart policy. Get it wrong and you leave a node down, capture bad state, or let old snapshots pile up. The payoff was operational: rebuilds that previously took weeks became restores measured in minutes.

Strip the domain away and the shape is common. Take the generic version, a backup scheduler. At any moment it looks at the world and picks one action from several: start a backup, delete an expired one, wait, or do nothing because we’re in a maintenance window. Which one wins is a matter of priority: maintenance beats a due backup, and a backup already running beats an expired one waiting to be deleted. That priority is the interesting part. It’s policy. And policy is exactly the thing most controllers end up hiding.

The mistake: making policy implicit

Here is the failure mode I want to name, because it’s more general than “nested ifs are bad”:

We make operational policy implicit by encoding it in control flow.

As policies accumulate, a reconcile loop tends to drift toward nested conditionals where precedence lives in the nesting order. It starts clean, with one if. Then a maintenance check wraps it. Then an expiry check nests inside that. A year later, the question “does maintenance mode suppress the expired-snapshot cleanup?” has an answer, but the only way to find it is to execute the nesting in your head. The policy is still there. It’s just implicit.

The cost is invisibility. You can refactor ugly code; you cannot review, exhaustively test, or explain a policy you can’t see when production does something surprising. Worse, that cost is paid repeatedly: every engineer who changes the system rediscovers the same reasoning, in review, during an incident, at the next requirement change.

So if control flow is the wrong place for policy, what’s the right one?

Three kinds of change

The answer is a decomposition. Three different concerns already exist here, and they change independently, at different rates:

• Observation is what you read from the world. It changes when the platform does: a new API, a new field, a different way to tell whether a node is running.
• Policy is the rules and their precedence. It changes when requirements do: a new retention requirement, a new “don’t touch anything during maintenance” rule.
• Execution is how you act, and how you log. It changes with the infrastructure: a different snapshot API, a new metrics backend.

These don’t move together. Policy churns while observation holds still; the platform shifts while policy stays put. Fuse them into one reconcile function and every change touches everything: a new requirement means editing code that also reads the cluster and calls the API. Things that change for different reasons should live in different places, so read the world, decide what to do, then do it, as three distinct things. None of that is new, and it isn’t the point yet. The point is what the separation lets you do with the middle step.

Read, decide, act

Concretely: read the cluster into an immutable snapshot of the world, call it the State. A decision function turns that State into an Action and does nothing else: no API calls, no logging, no clock. An executor takes the Action and performs the side effects. The reconcile loop is just the seam: read, decide, execute, requeue.

The first version of this separation wasn’t enough. The decision still knew too much about its world: it took a context and logged as it decided. The change that mattered was making the decision independently evaluable: a State → Action transformation with no dependencies beyond its input.

The decision has a deliberately boring signature:

func (s *Strategy) DetermineNextAction(state State) Action

No context, no client, no I/O. That buys three concrete things:

• the policy runs without a cluster, so you can ask “what would we do in this situation?” in a test, or in your head;
• I can write a table of tests with no API server, covering combinations that would be miserable to reproduce against a real one;
• an incident becomes reproducible: capture the State that produced a bad decision, replay it, and you’re debugging the exact decision, not a Kubernetes environment.

These are properties good policy should have, and policy buried in control flow usually doesn’t. But separating the decision only helps if the decision itself is legible.

Priority is the policy

Here’s a slice of the backup policy the way it tends to grow, as control flow:

if state.InMaintenanceWindow() {
    return NoAction
}
if state.BackupRunning() {
    return Wait
}
if state.BackupExpired() || state.RetentionExceeded() {
    if !state.IsLastBackup() {
        return DeleteBackup
    }
    return Wait
}
if state.ScheduleReached() {
    return StartBackup
}
return Wait

Read top to bottom and the obvious precedence is clear enough: maintenance returns first, so it wins. Now answer the question the ordering hides: if the last backup is expired and a scheduled backup is due, does it start a new one? You can work it out (the expired branch returns before the schedule check runs, so no, it waits), but you had to trace the returns to be sure. That precedence is real, and it’s buried in the control flow.

Here’s the same policy as an ordered list:

var rules = []Rule{
    Case("Do nothing during maintenance").
        Matches(State.InMaintenanceWindow).
        Execute(NoAction),

    Case("Let a running backup finish").
        Matches(State.BackupRunning).
        Execute(Wait),

    Case("Keep the last backup, even if it's due for cleanup").
        Matches(And(State.IsLastBackup, Or(State.BackupExpired, State.RetentionExceeded))).
        Execute(Wait),

    Case("Delete a backup that is expired or over retention").
        Matches(Or(State.BackupExpired, State.RetentionExceeded)).
        Execute(DeleteBackup),

    Case("Start a scheduled backup").
        Matches(State.ScheduleReached).
        Execute(StartBackup),
}

That same question is now answered by reading the list: “Keep the last backup” sits above “Start a scheduled backup”, so a last, expired backup is kept, not restarted. Priority is the order: it’s right there on the page, not something you have to execute to discover.

The rules are in priority order: the first that applies decides. Conditions compose: “delete a backup that is expired or over retention” is Or(BackupExpired, RetentionExceeded), one condition rather than two branches, and “keep the last backup, even if it’s due for cleanup” is And(IsLastBackup, Or(BackupExpired, RetentionExceeded)). Not you reach for far less: an ordered list puts an exception in a higher rule, not a negated guard on a lower one. The keep-the-last-backup rule sits above the delete rule, so the delete rule needs no Not(IsLastBackup), the guard the control-flow version above needed exactly. Why not a map of condition to action? Because a map throws the order away, and the order is the part that encodes precedence.

Make the decision explain itself

Reading the policy is the first payoff. The one that matters more, and that I rarely see discussed: because every rule is a named thing, the decision has a reason you can point at.

Walk the list to decide and you know which rule fired, which ones didn’t, and why. So log that:

decision: DeleteBackup (rule "Delete a backup that is expired or over retention")
facts: maintenance=false running=false expired=true retention=false lastBackup=false scheduleReached=false

Most software logs actions. Operational software should log decisions.

That is the difference between “the controller deleted a backup” and “the controller deleted a backup because it was expired, nothing was running, and we weren’t in maintenance.” One tells you what happened. The other tells you why this, and not the other things it could have done.

“Observability” is easy to wave at, so be precise. Think of it as a ladder:

• Correct: the right decision for the cases you anticipated.
• Testable: you can check the decision before production, not only in it.
• Understandable: a human can reason about why a rule exists.
• Observable: the running system tells you why it chose what it chose.

Correct, testable, understandable, observable. Each rung buys more operational confidence than the last: a test proves a decision is repeatable, but only observability tells you why a particular one happened in production. The ordered policy exists partly because that last step needs named rules to point at.

Reviewing policy, not control flow

The property that helps in an incident also helps at a far more mundane moment: code review.

Take a change that will actually happen: maintenance mode should no longer suppress backups. In the control-flow version, that’s a diff a reviewer has to mentally execute: delete the top guard and check that nothing below it relied on maintenance returning first. In the ordered version it’s a one-line change, moving or dropping the maintenance rule. The reviewer reads a policy change as a policy change, without tracing any control flow.

That’s an underrated design property. In long-lived systems, a surprising fraction of engineering time goes into optimising the team’s reasoning, not the compiler’s. And the easier a policy is to review, the less institutional knowledge the code silently demands. Legible policy is how a system stays changeable by people who weren’t there when it was written.

Where this pays off

The same design pays off elsewhere:

• Onboarding. A new engineer reads the policy as policy, instead of inferring it from control flow.
• Testing. A pure State → Action decision is straightforward to table-test: enumerate meaningful combinations, assert the action, no cluster required. Time-based rules stay deterministic by injecting the clock instead of reading the wall.
• Changing policy. This is the real test. When operational requirements change, you change exactly one rule, in one place. Policy changes on its own schedule, and a representation where that’s a one-line edit is one that respects it.

Choosing the representation

By now a skeptical reader has a list of “why didn’t you just…?” questions, which are really the same question as “when should I not do this?” So let me end there, because the honest answer is more useful than the pattern.

Isn’t this just a decision table? Yes. That’s the point. It’s a decision table with composable conditions and a name on every row, which a bare switch or a fixed column-matrix doesn’t give you.

Why not a plain switch? For a handful of cases, do. A switch is fine until precedence starts hiding in case order and conditions start wanting to compose. Switches aren’t bad; they just stop expressing policy once precedence and composition are the problem.

Why not an explicit state-machine library? Because this problem isn’t transition-heavy. A state machine answers “given where I am, what happens next?” This problem asks a different question: “given many independent facts about the world, which concern wins?” Ordered policy models that directly; a transition graph doesn’t.

Why not reconcile phases? Reasonable for some controllers; they just didn’t fit a problem whose essence is precedence among composable conditions.

And the real boundary, plainly: three conditions? Write the switch. Five states? Probably still the switch. Independent operational policies with explicit precedence, that several people will change over years? That’s when I start reaching for a representation that makes the policy first-class. It buys nothing in line count: an ordered policy usually runs longer than the nesting it replaces. What changes is where the decision lives, somewhere you can read, test, and log it. None of the pieces here (separating decision from I/O, ordered rules, decision tables) are new. The judgment is knowing when a problem has enough policy in it to be worth making that policy explicit.

The broader lesson

The specific thing in this article is an ordered policy for a Kubernetes controller. The lesson underneath isn’t about Kubernetes, or Go, or operators.

Whenever software makes operational decisions that matter, the decisions themselves become part of the system’s behaviour, as much as the actions they trigger. A deployment orchestrator choosing whether to roll forward, a scheduler placing work, a payment workflow deciding whether to retry or refund: same shape, same question. Treat those decisions the way you treat the actions: explicit, reviewable, testable, and observable, visible to the reviewer, the operator, the future maintainer, and you at 3 a.m. Because a system’s decisions are part of its behaviour. If the people who maintain and operate it can’t reconstruct why those decisions happened, it can’t really be operated.

Companion repository: the representation in isolation, on a toy backup scheduler.