How to Build a Supervised Harness for AI Development

AI tools write code very fast today. That is not the problem any more. The problem is different: can you trust the code that arrives on your screen?

I wanted to run two or three development tasks at the same time. Not more autonomy — more safety per task. So I designed a small system called flow. In this post I explain the design and the decisions behind it.

The real problem is your attention#

Many people think the bottleneck is the speed of the model. It is not. The bottleneck is you.

If a system works fast, but you must read the complete diff to trust it, you cannot work in parallel. You only become a full-time reviewer of code that you did not write. That is not productivity.

So the design has one central idea:

Every task must arrive at your review with a verdict that was not written by the same agent that wrote the code.

Everything else — logs, tickets, CLI, worktrees — exists only to support that sentence.

First decision: what NOT to build#

This part saves half of the work.

A workflow (steps controlled by code) is better than an agent (the model decides its own next step) when the path is predictable. My path is predictable: requirement → spec → development → validation → review → commit. It is a chain of prompts with gates, not an agent.

So I decided not to build:

• Autonomous multi-agent orchestration. The path is fixed. An agent that chooses the next step only adds cost and random behaviour.
• A FastAPI service with a database. Files plus git cover 100% of the need. Nothing to deploy, nothing to migrate.
• An LLM that decides if a task passes. The decision is an exit code.
• A small local model writing code. It has a clear place in the system, but not that one.
• Automatic writing in the company board. That is a shared tool. Visible, irreversible actions never happen without my approval. There is also a useful rule for maintenance: every component of a harness contains an assumption about what the model cannot do alone. When models improve, the assumption dies, and the component becomes dead weight. I review this list every few months.

The architecture in four layers#

The system has only four layers:

1. The ledger — files and git. The state of everything.
2. The transitions — a CLI called flow.
3. The gates — scripts that say pass or fail.
4. The cockpit — one command that shows me the board. Nothing more.

Layer 0: the ledger#

I wanted observability: all agents writing logs. But there is a rule that is not obvious here:

The log is written by the harness, not by the good will of the agent.

If you ask an agent to write logs, it forgets exactly when the task becomes difficult — and that is the moment when you need the log. The solution: every state change goes through the CLI, and the event is saved as a side effect of that change. Notes written by the agent are extra context, never the real trace. Git is the second trace, independent and hard to fake.

The structure is simple:

.flow/
├── index.json          # the board: all tasks and their state
├── config.yaml         # paths, concurrency limits
└── tasks/UP-412/
    ├── task.json       # machine-readable state
    ├── contract.json   # executable acceptance criteria
    ├── events.jsonl    # append-only log of everything
    ├── progress.md     # human handoff between sessions
    ├── spec/           # spec.md · design.md · tasks.md
    └── artifacts/      # test output, reviews, snapshots

Why JSON and not Markdown#

This is a small detail with a big effect. Models modify JSON files by mistake much less often than Markdown files. The state of the flow lives in JSON exactly because I do not want an agent to rewrite it "to make it cleaner".

Inside task.json there are two fields that I like a lot:

• A boolean "passes": true | false for every gate and every requirement. A boolean is auditable. The word "done" in prose is not.
• A counter of attempts per state. This is my loop detector: if the validation step fails three times, the system stops and calls me, instead of trying again. An agent that retries while stuck is the most expensive way to waste time.

The event log#

One line per event, append-only, never rewritten. Each line has a timestamp, the task, the actor, the state, the event type and a short summary.

The actor field always has a prefix: human:, cursor:, ollama:, claude:, script:. Why? Because the question I ask the log most often is: "did a person, a program or a model decide this?" With a prefix, the answer is a grep, not an investigation.

I also use a closed vocabulary of event names. If a name is not in the list, it is not saved. Without this rule, every session invents its own taxonomy and the log stops being useful.

Layer 1: the state machine#

The flow is a line of states: ingested → triaged → specified → contracted → developing → verifying → reviewed → committed → closed.

Every arrow is a new session, and its complete input is files. Nothing travels inside the memory of the model between states.

This is intentional. Between context reset and compaction, I chose reset. It removes "context anxiety" — the model closes the work too early when it sees the limit coming — and the price is orchestration. But I already have the orchestration: the CLI. So the reset is free. There is also a nice side effect: if a session dies in the middle, nothing is lost. Every session is "the engineer from the previous shift", and it only inherits what is written.

The missing piece: the contract#

My spec-driven workflow was already good, because the spec is the source of truth and every requirement has a scenario. But one step was missing: before writing one line of code, the implementer must declare which concrete check will prove each requirement. Then that is frozen.

Without this step, the classic failure appears: the success criteria is negotiated after seeing the code. That is the same as having no criteria.

So the contract is a small JSON file: each requirement, a short statement, and a command with an exit code. It also has two extra fields:

• spec_sha connects the contract to the version of the spec. If the spec changes, the contract is invalid and must be written again.
• out_of_scope lists the paths that this task must not touch. It is my safety net against scope creep. If a requirement cannot be expressed as a command, the process stops and asks me. That is not a bug, it is the point.

Layer 2: gates, where no model decides#

The command flow verify runs, in order, and stops at the first failure:

1. lint
2. typecheck
3. tests
4. the checks of the contract
5. diff hygiene The verdict is the exit code. This is important: the script executes, the exit code judges. A skill that gives an opinion about the tests brings back exactly the bias I want to remove.

Diff hygiene: the most profitable gate#

It is a script of about 100 lines, and it saves more problems than all the rest together. The reason is simple: the easiest path to a green test suite is to delete the red test. You cannot fix that with a polite prompt. You block it with a script.

The verification fails if the diff contains:

• deleted or renamed tests without a reason in the spec
• new skip, xfail, early return or assert True
• commented asserts
• files outside the scope of the contract
• coverage below the limit
• possible secrets or a versioned .env
• new TODO / FIXME in the touched code
• commits without a ticket reference

The blind reviewer#

This is the most important idea of the whole system. An agent that reviews its own work praises the result with confidence, even when the quality is clearly poor. The worst version is subtle: it finds the real problem, and then it convinces itself that the problem is not important.

The only strong solution is to separate the generator from the evaluator. So the review happens in layers, from free to expensive:

Layer 2 runs only if 0 and 1 pass. And its design is what makes it cheap and reliable:

• It sees only the spec, the contract, the final diff and the test output.
• It never sees the development conversation or the failed attempts. A reviewer that knows why the author did something tends to accept the explanation. Blind, it can only compare the result with the contract.
• The verdict is binary and hard. If any check is not proved, the task fails. No "acceptable with comments".
• It returns concrete bugs, not feelings. A bug without reproduction steps does not count. One warning: the reviewer out of the box is too soft. Making it strict is an explicit process. I save every review, and when I find a problem after the review passed, I treat it as a failure of the reviewer. Then I read that review, find the exact moment where the criteria became soft, and update the prompt. The reviewer prompt is the asset that grows most in value.

Layer 3: safe parallelism#

This is the part I really wanted. Two rules and one tree per task.

One git worktree per task. Two sessions never share a working tree. Without this, parallelism is a source of silent corruption: one session saves a file that the other was editing, and the result belongs to nobody.

One isolated stack per worktree. With Docker Compose and fixed ports, two worktrees collide in project name, ports and volumes. So every worktree gets its own project name, and the ports come from a hash of the task id: stable between sessions, with no central registry.

A concurrency limit of two. This limit is not technical. It is my capacity to supervise. The way to get more real parallelism is not to raise the number — it is to make the gates so good that every task arrives in a state that is almost ready to merge.

Where each tool belongs#

I use three different models, and each one has a clear job:

• Cursor writes the code. My company pays for it, and it is the tool with the best context of the repository.
• Claude reviews, once per task, blind. It is the point of the pipeline with the most value per token, so it is the best place to spend my subscription.
• A local 9B model classifies and summarises. This is the routing pattern: send the easy work to a small model and the difficult work to a big one. The small model is good for: classifying the ticket type, estimating complexity, detecting missing information, summarising the diff in ten lines, and writing a draft comment that I approve before sending.

The small model must not: write production code, judge if the code follows the spec, decide state transitions, or write specs. The gate that needs most rigour is the gate with least space for a small model. And a weak spec contaminates everything below it.

Build the gates before the automation#

The order matters. If you automate the entry of tasks before you can trust the exit, you only fill a pipeline that you do not trust.

So phase one is the skeleton: the file structure, the CLI, verify with hygiene, worktrees, and manual ticket entry. With only this, I can already run two tasks in parallel with a safety net. It is the phase with most value per line of code.

Phase two adds the rigour: the contract state and the blind reviewer. Phase three adds the automation of the ticket intake and the local model. Phase four is optional reports and metrics.

What I learned#

Three ideas stay with me after this design:

1. The verdict must come from a different place than the code. Everything else is decoration.
2. A gate is a script, not a prompt. If you can express a rule as an exit code, do it. A rule inside a prompt is a suggestion.
3. Every component encodes an assumption about the model. Write that assumption down. When it stops being true, delete the component. A harness that only grows becomes the new problem. And one limit never expires: the concurrency cap. Models will improve, gates will become unnecessary, prompts will change. But my attention will always be the real limit of the system.