By The Chiri Team
An agent with the ability to finish a job will still stop early. It stops when nothing tells it what finished means. This is the finding at the center of a Factory research report, published in August 2026. It changes how a coding agent should be built.
Factory ran 24 tasks from ProgramBench, a benchmark built from large real software projects. The company compared two setups. In the first, a single agent took a task from a written spec straight through to a finished output. In the second, three separate roles split the work: an orchestrator, an implementer, and an independent validator.
The validator role is the finding. Before the implementer writes any code, the validator builds the completion standard and an independent test instrument. The validator does not see the implementer’s approach in advance. The implementer does not see the validator’s test cases. Once the standard is set, it cannot be weakened later in the task.
How the three roles divide the work
Each role in Factory’s system carries a separate job. Each job stays separate through the whole task.
The orchestrator manages the task from the outside. It sequences the work and coordinates between the other two roles. It does not write the implementation or the test cases itself.
The implementer writes the code. It works from the same written spec the validator saw. It does not see the validator’s test suite, and it does not see the comparison rules the validator built.
The validator does one job only: define what a correct, finished answer looks like, then build the instrument that checks for it. This work happens before the implementer starts. The validator’s standard does not change once the implementer begins.
A single agent collapses all three jobs into one process. It plans its own work, does the work, and checks its own work against a standard it set for itself. That check often happens after the fact. Nothing in that loop forces the agent to define “done” before it starts. Nothing stops it from adjusting the definition once the work is already underway.
The gap between ability and completion
Across the 24 tasks, the three-role system produced large gains over the single agent. The gain held across three different models. Median scores moved from 56.7% to 89.3% on Fable 5, a gain of 32.6 points. On Kimi K3, median scores moved from 45.1% to 75.4%, a gain of 30.3 points. On GPT-5.6-Sol, median scores moved from 48.6% to 66.2%, a gain of 17.6 points.
None of these three models changed between the two setups. The same model ran as a single agent in one condition. It ran as one of three coordinated roles in the other. The gain came from splitting the work, not from a stronger model doing the work.
One task inside the set makes the pattern concrete at a larger scale. The task asked the agent to reproduce a large geospatial library, GDAL. The single agent reached 36% behavioral parity with the reference program. The three-role system reached 90% on the same task, a gap of 54 points.
Factory’s own conclusion is direct. The agent had the coding ability. It stopped early because nothing defined what “done” meant. The gap was not skill. The gap was a missing standard.
This distinction carries weight. A capability gap calls for a better model or a longer training run. A completion-standard gap calls for a different process. In that process, the definition of finished exists before a single line of code is written. That definition cannot move once work is underway.
What the validator actually builds
The validator’s output is not a single check. Factory describes it as building “a weighted body of cases and the comparison rules needed to judge the candidate’s output.” This happens before implementation begins. The validator does not hand this body of cases to the implementer.
That separation matters mechanically, not as a trust exercise. An implementer that can see the test suite in advance can write code that satisfies the letter of each case. That code can still miss the underlying behavior of the reference program. An implementer that cannot see the suite has to build the real thing.
The validator also cannot lower the bar mid-task. Once the completion standard is set, the task cannot redefine “done” as “whatever the implementer managed to produce.” This closes a loophole. A single agent can otherwise narrow its own scope quietly, until the remaining work looks finished.
A gap Chiri found in its own product
The pattern was not only visible in a benchmark. Chiri’s engineering team found a version of the same gap inside its own platform.
An internal agent had assumed that agent analytics existed as a feature. The feature was not actually built. The gap sat undetected because no validator existed to check the assumption against the working product. Nothing defined “done” for that piece of work, so nothing caught that a core piece of it was missing.
The failure did not come from a lack of coding skill. It came from the same gap Factory documented in its benchmark: an agent working without an independent standard for what “done” required. A missing feature passed as present.
The discovery drove a change to how Chiri writes specs internally. A validator is now written as part of the spec, before any code exists. It gets written in the same step where the task itself gets defined. Chiri holds validator coverage to a minimum bar, the same way test coverage gets held to a minimum bar on a mature codebase. A spec without a validator does not count as a finished spec.
The cost argument for writing the check once
A resource argument sits underneath the process argument. A validator is written once, and that step costs tokens. After that, the validator runs as ordinary deterministic code. It does not need a model to execute it again.
The alternative is to keep spending tokens on a model. That model re-checks the same kind of output every time a new task runs. That cost repeats on every task and every run, indefinitely. A deterministic validator pays its cost once, then runs at no further token cost.
This is a routing decision, not a capability decision. Some checks need a model’s judgment. A check for whether a program behaves like its reference implementation does not need reasoning at every step. It needs a rule, written once, applied by ordinary code from then on.
The distinction scales differently over time. Token spend on repeated model checks grows with every task run. Token spend on writing a validator grows with the number of validators written. It does not grow with how many times each one runs afterward. On a team running many tasks against a stable set of standards, that difference compounds fast.
The open question
The validator model raises a question it does not answer. Something has to verify that the first validator is correct.
If a validator is wrong, it can pass broken work or fail correct work. Either failure mode is costly. A wrong pass ships a defect behind a passing check. A wrong fail burns implementer time chasing a standard that was never right to begin with.
Factory’s research does not resolve this, and no clean answer exists yet. A second, independent validator could check the first one. That second validator needs its own standard for correctness, and the question repeats one layer down. At some point, a person has to set the first standard and accept the risk that comes with it.
This deserves a plain statement rather than a smooth-over. The three-role system solves the problem of an agent that stops early. It does not solve the problem of who checks the checker. Any team that adopts this pattern inherits that open question along with the gains.
Why a pod outperforms a single larger model here
The Factory result lines up with a broader pattern. Multi-role systems perform differently than a single larger model on hard tasks. A single agent, however capable, carries every role itself. It defines the task, does the work, and judges its own output. Nothing outside that one process pushes back on it.
A three-role pod splits those functions across separate agents with separate information. The orchestrator manages the task. The implementer writes the code. The validator defines success on its own and holds to it. No single role can quietly redefine success to match what it already produced.
The GDAL result shows the size of that difference on one large real program: 36% against 90% on the identical task. The gap did not come from a stronger model inside the three-role system. Factory used the same set of models in both setups. The gap came from structure.
A bigger single model still runs all three functions through one process. It can get better at each function individually. It still plans, builds, and judges its own work, with no outside check on any of the three. A pod of smaller, specialized roles removes that single point of failure by design. Factory’s data shows the removal working across three different underlying models.
This lands differently depending on where you sit
For an engineering leader evaluating agent tooling: the benchmark gain is not about model choice. Fable 5, Kimi K3, and GPT-5.6-Sol all improved under the three-role structure. The structure is the lever, not the model swap.
For a team building agent workflows in-house: write the validator before the implementer starts work. Keep the two roles blind to each other’s output. A shared spec that both sides can see recreates the single-agent failure mode inside a multi-agent wrapper.
For a platform team weighing token spend: a validator built once, then run as deterministic code, is a fixed cost set against an ongoing one. Model-based re-checking on every run compounds over time. A rule written once does not.
For anyone setting the first standard: the open question about who verifies the validator has no clean answer yet. Treat that as a known risk to manage, not a solved problem to cite as settled.
For a team that ships without a validator today: the GDAL gap is the size of the risk a missing completion standard can carry on one large task: 36% against 90%. A spec without a validator is a spec without a definition of done.
What would it take to trust a validator someone else wrote?