A single prompt left the codebase broken in four runs out of five

I published a version of this article that said agent loops change nothing. A reader told me that the best engineers he knows all run loops, so either they are wrong or I am.

He was right. My experiment was built badly, and this is the corrected version.

The flaw was not in the measurement, it was in the task. Every task I benchmarked was small enough for the model to finish in one shot, so the loop checked its stop condition, found the tests already green, and exited. I then reported that loops almost never loop. That is a fact about my tasks, not about loops, and I should have seen it before publishing.

So I built a task big enough to actually need one. Ten interacting requirements, ten to twelve files touched. Here is what one prompt does with it:

In four of the five single-prompt runs the resulting code did not even import. The test runner failed at the first line of the file. The model worked for two minutes, produced a large diff, and left the repository broken.

That is the regime where loops live, and I had never tested it.

What the first version got wrong

Two mistakes, worth naming because they are easy to repeat.

The tasks were too small. Fixing one function, adding one feature, migrating six files. A current model finishes those in one pass, so there is nothing for a second pass to do. Measuring a loop on work that does not need one tells you nothing about loops.

The metric hid the effect. I reported hidden-suite pass as a binary: all twelve tests or nothing. On small tasks that is fine, because runs either work or do not. On a big task it erases everything interesting. By the binary metric, all three arms score 0/5 on the task below. By the honest metric, counting how many of the twelve hidden tests pass, the single prompt scores 1.4 and the loop scores 8.0. Same data, opposite conclusion.

A binary pass rate on a hard task measures whether you cleared the bar, not how far you got. When nobody clears the bar, it reports a tie between a run that broke the build and a run that got 9 of 12 requirements right.

The task that was big enough

Ten requirements on a small pricing library, specified in full and designed to interact: fix a cent-dropping split including negatives, migrate every amount to integer cents, validate currencies, VAT rounded half away from zero, B2B reverse charge, line-level discounts, integral percent discounts, promo codes with stacking and expiry and minimums, an invoice splitter, and input validation.

A reference solution passes 10/10 visible and 12/12 hidden, so the task is solvable. The untouched base fails it.

Three arms, five runs each, on Haiku 4.5, which is deliberately not the strongest model available. A model that one-shots the task puts you back in the regime where loops do nothing, and that regime I had already measured to death.

Single prompt is one invocation. Loop re-runs the same prompt with fresh context until the visible tests pass, capped at eight passes. Engineered loop implements what the Loop Engineering paper describes as the building blocks of a real loop: a planning pass that writes a checklist to PLAN.md, one item per pass instead of the whole task, a verifier subagent that reads the spec rather than the tests and runs with Edit and Write disabled so it physically cannot touch the code, a pass budget, and a stall detector that stops when two passes produce no change.

Every task carries a hidden test suite the agent never sees, covering the rules the prompt states but the visible tests do not check. The visible suite is the loop’s stop condition. The hidden suite is the grade.

Results

Three things in that table.

The loop works, and the effect is large. One prompt passes the visible suite once in five attempts. The loop passes it five times out of five. A two-sided Fisher exact test on 5/5 against 1/5 gives p = 0.048, which with five runs per arm is about as much significance as this design can produce. On the graded hidden score the loop takes a run from 1.4 correct requirements to 8.0.

The single-prompt failures are not near misses. Four of five left code that cannot be imported. The loop’s contribution here is not polish, it is the difference between a broken repository and one that builds and passes its tests.

The engineering did not pay for itself. The engineered arm scored 7.8 against the naive loop’s 8.0, took three and a half times longer and cost nearly four times more. One run collapsed outright at 2 of 12. All five hit the stall detector. I expected the verifier subagent to be the component that mattered, and on this task it was not.

What no arm fixed

Not one run in fifteen passed all twelve hidden tests. The residual failures are concentrated and consistent: the promo code rules failed in 5 of 5 runs in both loop arms, and the pipeline composition order failed in most.

Those rules are written in the prompt. “A code is expired when now >= expiresAt, so a code expiring exactly now does not apply.” The visible tests did not check it, so it did not get built, and nothing in the loop noticed. The verifier subagent, explicitly told that a green suite is not evidence a requirement is met, signed off anyway.

So the part of my original finding that survives is narrower and more precise than what I published: a loop raises the floor and does not raise the ceiling. It reliably gets you from broken to working. It does not get you from working to correct, because nothing in the loop knows what correct means except the tests you wrote.

That matches the literature better than my first version did. DeLoop finds runs that fall into repetition fail 59.5 percent of the time against 33.4 percent for runs that do not, which is why the stall detector earns its place. Prompt-Induced Waste finds harness choice moving cost per success by 5 to 30 times, far more than prompt wording. The loop is the harness. It is the lever.

And Huntley, who published the technique in July 2025, has been saying the quiet part all along: “engineers are still needed”, and there is “no way this is possible without senior expertise guiding Ralph”. The loop gets the work done. Knowing what done means is still the job.

Pros and cons, from 95 runs

What I do now

Match the loop to the task size, and stop arguing about it in general. Small task, one prompt. Task where the first attempt plausibly leaves the build broken, loop. That boundary is the entire decision, and my original article got it wrong by only testing one side of it.

Spend your effort on the stop condition, not on the loop machinery. The engineered arm added planning, review and escalation for four times the cost and did not move the hidden score. The thing that moved it was having a loop at all. The thing that would have moved it further is tests that encode the requirements, which no amount of orchestration substitutes for.

Write the assertion for every rule you care about before you start. Every residual failure here was a sentence in the prompt with no test behind it. The agent reads your test suite as the specification, whatever your prompt says.

Limits

Five runs per arm, one model, one codebase, one big task plus the four small ones from the first version. The epic-tier comparison reaches significance on the visible suite and nowhere else. The engineered arm is my implementation of a published description, not anyone’s production setup, and a better one may well beat the naive loop where mine did not. Costs are computed by the CLI from list prices on a subscription, so they are a conversion, not an invoice. Ninety-five runs, $16.34 equivalent.

The harness is not published, so take the numbers as one person’s measurement rather than something you can rerun as it stands. The design is described above in enough detail to rebuild: a base library, a visible suite that doubles as the loop’s stop condition, a hidden suite copied in only after the agent has finished, and a reference solution proving each task is solvable before any run.

The takeaway

I benchmarked loops on tasks that did not need loops, used a metric that could not see partial progress, and published that loops do nothing. A reader pushed back, I built the task I should have built first, and one prompt left the codebase unable to import in four runs out of five.

The loop is not a quality multiplier. It is a floor, and the floor turns out to matter enormously: it is the difference between a broken repository and a working one. What it is not is a substitute for knowing what you asked for, which is why the twelfth hidden test still fails and why the people getting the most out of loops are the ones writing the stop conditions by hand.