Same outcome, not the same output

Someone on X asked a question that sounds like a job interview trap:

LLMs are non-deterministic. So how can an AI agent be idempotent? If the model can change its response, how can we guarantee the same outcome?

My answer was short: you don’t. You make the parts that matter deterministic.

I run an autonomous coding loop on Tallyroot, my personal finance app. It picks an issue, writes the fix, gets reviewed by another agent, and merges to main if everything passes. Give it the same issue twice and you get two different diffs. That’s fine. What has to be the same is what lands on main, and what it had to get through to land there.

The numbers

Rounds logged since August	653
Rounds on the day I wrote this	7
One of those rounds, at API prices	$7 to $27
PRs merged by plain code since September 29	108
Guardrail tests	151
Lessons in the playbook	222

It all runs on a Claude Max subscription, so nothing here was actually billed. The dollar figures are what the same tokens would have cost at API prices.

What’s allowed to vary

Almost everything an agent does. Which file it opens first, how it names things, how many turns it takes, what the reviewer nitpicks. Seven rounds in one day ranged from $7 to $27. If I needed any of that to be repeatable, I’d have lost already.

So I stopped asking the model for the same output. I ask the system for the same outcome: nothing reaches main unless it passed the same checks as everything else.

What isn’t

The parts that decide are plain code.

Every agent answers in a fixed schema, so the script gets fields back, not prose it has to interpret. Before any agent starts, code checks the input: a pinned issue has to be open, not an epic, not waiting on something, and without an open PR.

And since the loop left Claude Code at the end of September, the merge isn’t an agent at all. The agent can only say a PR is ready, and which commit it tested. Then the runtime checks that the commit hasn’t moved, checks that GitHub is green, runs the whole test suite again in a fresh checkout, and merges exactly that commit. Taking my agent loop out of Claude Code has the details.

The model can be as creative as it likes up to that point. The gate doesn’t care how the code got there.

Idempotent where it counts

The question was about idempotency, and in the loop it lives in the commit.

A refusal belongs to a commit. If the gate refuses a commit, the loop never tries that commit again. Only a new commit clears it. Otherwise the loop could rerun a flaky suite until it happens to pass, which is a quiet way to merge something broken.

Retries work the same way. A step only gets retried if the agent never started: zero turns, so a retry can’t do anything twice. An agent that already pushed a branch or opened a PR is never started again from scratch. If it dies halfway, a later round picks the PR up where it stands.

I had a good example the day I wrote this. A round built a fix, and the review passed. Then the merge step died partway through, on an API overload. The loop didn’t retry it, because it couldn’t know what had already happened. The next round resumed the PR, applied six small review fixes, and sent it to the gate. My internet dropped out while the tests ran, a few database tests hung for 17 minutes each, and the gate refused the PR for running past its 30-minute limit. Nothing merged. Afterwards I ran the same test command on the same commit: 7,129 tests passed, and I merged it.

Two rounds, an overload and an outage. Same outcome: that commit on main, with that suite green.

The ratchet

The second half of my answer was that every bad answer that gets close becomes another guardrail.

When a mistake slips through review, or almost does, the loop doesn’t only fix that one. It adds a lesson to the playbook every agent reads. If the mistake can be checked mechanically, it also adds a test that fails when the mistake shows up again. That’s where the 151 guardrail tests came from.

One of my favorites is .rowCount. Tallyroot talks to Postgres through a library called postgres.js, which reports how many rows a write changed as .count. The more popular library, pg, calls it .rowCount, so that’s what an agent writes from habit:

const result = await db
  .update(workspaceApiKeys)
  .set({ revokedAt: new Date() })
  .where(/* ... */)

return (result as unknown as { rowCount: number }).rowCount > 0

On postgres.js, .rowCount doesn’t exist. It’s undefined, and undefined > 0 is false. The write still runs, so nothing crashes, and every “did it change a row?” check answers no. It showed up twice in one sweep. Revoking an API key worked, and the app said it had failed.

Fixing those two places was easy. The guardrail is what stops the third: a test that reads every file in the backend and fails the build if .rowCount appears anywhere outside a comment. It doesn’t run the code or ask anyone’s opinion. It reads the source, and it doesn’t care how convincing the PR description is.

The ratchet only turns one way. That’s the point, and it’s also the catch. In the first post I wrote how the share of merges that were the loop’s own machinery went from 35% to 62%, because nothing ever turned a pile of guardrails back into one simpler rule. A ratchet needs a way to consolidate, or the guardrails become the product.

What stays random

Two runs on the same issue still give different diffs, different costs, sometimes a different approach. That’s fine. It would be the same with people.

Give the same issue to two engineers and you get two solutions. One extracts a helper, the other fixes it in place. One writes three tests, the other writes one that covers more. Often neither is better, just different, and both do the same thing. Heck, even the same developer would come up with a (maybe not so) slightly different solution on a different day. Nobody calls a team non-deterministic for that. We review the change and run the tests, and if they pass, it ships.

That’s all the loop does too. What matters is that both versions had to pass the same gate, and that a version that can’t pass never lands.

The model isn’t predictable. The way its work reaches main is.

Or, the way I put it on X afterwards: you aim for the same outcome, not the same output.

If you’d like to run something like this on your own project, I’m packaging it up. Join the waitlist below and you’ll get one email when it’s ready.