Diagnose the harness gap: measure before you scaffold: Practical Field Guide

On this page

01Working definition
02Field situation
03Worked example
04Build it, with checkpoints
05Operating context
06Hands-on lab
07Failure clinic
08Production boundary
09Sources and claim limits

Learning objectives

Separate model capability failures from missing context, state, control, and feedback
Capture a reproducible baseline with quality, intervention, scope, cost, and recovery evidence
Choose the smallest harness investment supported by the observed failure
State the limits of a single before-and-after run without turning it into a benchmark claim

Before you start

• A local Git installation and a coding agent that can read and edit a sandbox repository
• The Release Desk starter fixture with no real credentials, customer records, or production connections

Working definition

Diagnose the harness gap

A harness gap is a repeatable failure caused by what surrounds the coding model: unclear intent, missing repository knowledge, an unhealthy environment, lost work state, weak boundaries, or feedback that cannot prove the result. Diagnosis starts with a controlled run because scaffolding added before evidence can hide the actual problem and create maintenance cost of its own.

Teams often react to a poor run by adding a longer prompt or a more capable model. That can improve one attempt while leaving the repository unable to explain its architecture, start reliably, resume after interruption, or show an end-to-end result. The next task fails for the same structural reason under a different surface symptom.

A baseline changes the discussion from confidence to evidence. You record what the agent saw, what it changed, the first moment it claimed completion, which checks actually passed, and how much human repair followed. Each later harness layer must improve one of those observations or be removed.

Field situation

Release Desk baseline run

Release Desk is a named synthetic course application. Its defects and run receipts are teaching fixtures, not reported customer outcomes.

Owner: A technical lead deciding how much autonomy a coding agent has earned in a small brownfield product.
Decision: Which observed failure deserves the first harness control, and which tempting scaffolding has no evidence yet?
Starting state: The application can be installed with undocumented local knowledge. Unit tests cover request parsing, but no repository map, session state, browser acceptance, or run receipt exists.
Expected outcome: A baseline receipt that a second reviewer can replay against the same commit and acceptance cases without reading the chat transcript.

Constraints

• The agent may edit only the sandbox clone and may not access external accounts or network destinations
• The run stops at 45 minutes, a destructive request, an unclear product decision, or the agent's first completion claim
• No harness file may be added until the baseline receipt is saved

Worked example

A green unit suite and a broken review flow

Evidence status: Named synthetic scenario

The agent implemented a request-changes API branch, added unit coverage, and reported success after the test command passed. It also reformatted an unrelated status component. In the running app, the reviewer note disappeared after refresh and the audit event was inserted twice after a simulated timeout.

The lead did not add a longer instruction file. The receipt classified the lost note and duplicate event as missing end-to-end feedback plus an unbounded retry effect. The unrelated formatting change was recorded as weak scope control. Startup happened correctly, so environment automation was not the first intervention.

The next experiment adds a browser acceptance case and an idempotency fixture while keeping the model, task contract, and repository commit stable. This produces a falsifiable claim: the new controls should catch both seeded failures before completion can be reported.

Limits

One synthetic run does not estimate general model reliability. It identifies concrete failure paths in this repository and provides a baseline for controlled course experiments.

Method

Build it, with checkpoints

Field situation

Which observed failure deserves the first harness control, and which tempting scaffolding has no evidence yet?

01Freeze the comparison
02Observe without coaching
03Run independent checks

Acceptance checks

The baseline will usually expose at least one gap between the agent's completion claim and the full acceptance pipeline. A valid result may also show that a proposed harness layer is unnecessary for this task.

Why this visualA five-layer diagnostic map lets learners assign an observed failure to a repairable system boundary before choosing a control.

01
Freeze the comparison
Create a branch from the course baseline tag. Save the exact task, environment versions, agent setup label, stop conditions, and starting time before the agent reads the repository. Do not include hidden tips or the solution in the task text.
CHECKPOINT · The receipt identifies one immutable commit, task fixture, setup label, and explicit stop boundary.
02
Observe without coaching
Let the agent inspect and work. Log every human clarification, permission change, manual command, and correction. When it first reports completion, capture the time and commit before offering any repair instruction.
CHECKPOINT · A reviewer can distinguish autonomous work, requested clarification, and human repair from the event log.
03
Run independent checks
Execute the public unit and integration checks, then follow the browser acceptance script from a fresh process. Record pass, fail, and not-run explicitly. A plausible diff or successful build does not replace a user-visible state check.
CHECKPOINT · Every acceptance case has evidence and no failed or skipped case is represented as complete.
04
Choose one intervention
Map each failure to the five-layer model and select the smallest next control. State the expected change in an observable metric. Leave unrelated improvements in a parking list so the next run still isolates cause and effect.
CHECKPOINT · The next control names one target failure, one expected signal, and one condition for removing the control.

Operating context

Beyond the demo

Hold the task and environment still

Use the provided request-changes feature, fixed seed data, pinned dependency lockfile, and one declared model setup. Save the starting commit and the exact task text. A later run may use a holdout task of comparable shape, but it cannot use a simpler acceptance path and still count as a useful comparison.

Do not tune the repository during the baseline. If installation fails, preserve that failure as evidence. If the agent asks for information that only exists in your head, write down the question but do not answer it until the run reaches the agreed stop. The missing information is part of the result.

Classify evidence, not personality

Map each observation to intent, legibility, continuity, control, or feedback. A wrong file caused by a missing architecture map is a legibility problem. A correct diff that breaks the browser flow is a feedback problem. Repeated writes after an unknown timeout point to control and recovery. Labels such as careless or confused do not identify a repairable system boundary.

Some observations remain ambiguous after one run. Mark them unknown and design the smallest follow-up experiment. For example, repeat the same run after adding only an executable health check. If startup waste disappears while feature quality remains flat, you learned where that layer helps and where it does not.

Hands-on lab

Run and score the unassisted feature task

Create a clean branch from the supplied baseline commit. Give the coding agent the exact request-changes task, observe without coaching, then verify the result with the public checks and the manual browser script. Record the first completion claim separately from the final repaired state.

Prepare

• Record the repository commit, tool and model label, local runtime versions, and wall-clock start time
• Confirm that the clone contains fixture data only and that network access is disabled or allow-listed to no external destinations

Deliverable

A committed baseline JSON receipt, the agent diff, public check output, one browser evidence capture, and a short decision naming the first control to test.

Starter kit: Baseline run worksheet

JSON

{
  "runId": "he-00-baseline-001",
  "startingCommit": "REPLACE_ME",
  "taskFixture": "request-changes-v1",
  "agentSetup": { "tool": "REPLACE_ME", "modelLabel": "REPLACE_ME" },
  "limits": { "minutes": 45, "externalWrites": 0 },
  "firstCompletionClaim": null,
  "acceptance": [],
  "humanInterventions": [],
  "filesChanged": [],
  "failureClass": [],
  "limitsOfEvidence": "single synthetic run"
}

Downloadable artifacts

Baseline worksheet

harness-baseline.json · JSON

An editable course fixture for the main lab. Save it inside the Release Desk repository before running the acceptance command.

Run receipt template

he-00-receipt.json · JSON

A compact evidence record for the check, environment, result, and limits that another reviewer must be able to inspect.

Acceptance command

npm run harness:baseline -- --receipt .harness/harness-baseline.json

Expected receipt

PASS he-00 baseline-recorded
task=request-changes-v1 evidence>=5 externalWrites=0
decision=recorded limits=single-run

Expected result

Carry forward

Keep the baseline receipt unchanged. Modules 01 through 09 will append comparison receipts, and the capstone will use the same dimensions on a holdout feature.

Acceptance checks

01The starting commit, exact task fixture, environment, limits, and agent setup are present
02First completion and final verified state are stored as separate events
03Quality, human intervention, changed scope, elapsed time, and recovery evidence are recorded
04Every conclusion points to captured evidence and includes the limit of a single synthetic run

What breaks

Failure clinic

F1The learner explains that the agent felt unreliable but cannot point to a failed case.

Inspect: Check for saved commands, outputs, browser steps, timestamps, and the first completion event.
Likely cause: Observation stayed in the chat and was summarized from memory after the run.
Repair: Repeat the run with the worksheet open and capture each event before interpretation.
Prevent next time: Require evidence fields before the decision field can be completed.

F2The second run improves, but several controls and a different model changed at once.

Inspect: Diff the starting commit, task, environment, setup, and harness files between receipts.
Likely cause: The experiment optimized for a good demo instead of isolating one harness effect.
Repair: Return to the baseline and change only the selected control before re-running.
Prevent next time: Store a comparison manifest and fail the check when undeclared variables drift.

F3A successful test command is recorded as full task completion.

Inspect: Compare the command's coverage with the task's API, persistence, browser, and retry criteria.
Likely cause: The easiest available check became a proxy for the whole user outcome.
Repair: Mark uncovered criteria not-run and execute the public browser and failure fixtures.
Prevent next time: Give every acceptance criterion its own evidence slot and terminal status.

Beyond the demo

Production boundary

01Baseline repositories contain no real secrets, identities, customer content, or production endpoints
02Task, model label, dependency lockfile, operating system, runtime versions, and starting commit are recorded
03Agent output, human intervention, deterministic checks, and browser evidence remain distinguishable
04A stop condition ends the run on destructive intent, unclear product judgment, time limit, or external write
05Receipts state their evidence limits and never present a single run as a general benchmark
06Every proposed harness layer has a target failure, observable signal, owner, and removal condition

Agents vs workflows →

Evidence status

Sources and claim limits

Sources support the named claims; they do not guarantee the same result in another system.

[1]
Harness engineering: leveraging Codex in an agent-first world
OpenAI · Public case · 2026-08-26
repository knowledge · agent legibility · mechanical enforcement · entropy management
[2]
Effective harnesses for long-running agents
Anthropic · Published research · 2026-08-26
initializer pattern · feature ledger · session handoff · end-to-end verification
[3]
Building effective agents
Anthropic · Official documentation · 2026-08-26
simplest viable architecture · workflow patterns · environmental feedback · stop conditions
[4]
Learn Harness Engineering
Walking Labs · Public case · 2026-08-26
project-based sequence · five harness subsystems · loop engineering · graph engineering
[5]
Andrej Karpathy's AI Engineering Playbook
AI Builder Club · Public case · 2026-08-26
Software 3.0 framing · spec-diff-eval practice · parallel session framing · repository instructions