Build the verification loop: grade behavior, evidence, and restraint: Practical Field Guide

On this page

01Working definition
02Field situation
03Worked example
04Build it, with checkpoints
05Operating context
06Hands-on lab
07Failure clinic
08Production boundary
09Sources and claim limits

Learning objectives

Translate acceptance criteria and prohibited effects into deterministic checks and reviewable evidence
Order fast structural checks, focused behavior tests, integration fixtures, and browser acceptance without hiding failures
Evaluate completion honesty, retry safety, scope discipline, and evidence quality alongside feature behavior
Lock a holdout set and compare baseline with candidate across repeated runs and reviewer burden

Before you start

• Modules 00 through 06, including the original baseline, locked task contract, typed tool observations, and mechanical rules
• A deterministic Release Desk fixture with browser automation, effect ledger, fault injection, and no production connection

Working definition

Verification and evals

A verification loop is the versioned path from a task contract to observable evidence and an allowed state transition. It checks structural invariants, behavior at the relevant boundary, failure and recovery cases, forbidden effects, and the integrity of the receipt. Completion is created by this verifier for a specific commit, environment, fixture, and contract version; it is not a confidence statement from the coding session.

Unit tests can pass while a user cannot reach the feature, persistence disappears on refresh, an audit effect duplicates on retry, or an unrelated file changes. Browser success can also hide skipped error cases and architecture drift. The harness needs multiple layers because each layer answers a different question and carries a different cost.

Evaluation provides the evidence needed to decide whether a harness improvement is real. Compare a fixed baseline and candidate on retained tasks, include repeated trials where model behavior varies, and report reviewer interventions, scope violations, completion errors, time, and cost. One successful demonstration is useful for debugging but weak support for scaling autonomy.

Field situation

Passing tests, duplicate audit effect

The tasks, candidate runs, model behavior, evaluator results, and reviewer timing are deterministic or seeded Release Desk fixtures.

Owner: An evaluation owner deciding whether the new repository harness supports a longer unattended task window.
Decision: What evaluator catches the duplicate effect, and what evidence is required before any autonomy increase?
Starting state: Unit, type, and API tests pass. In one of five unknown-outcome trials, retry creates two audit records. The browser displays the correct final state, so a happy-path evaluator still reports success.
Expected outcome: The candidate fails the critical retry gate, the trace identifies the missing reconciliation boundary, and a repaired version must pass repeated development and holdout runs before a scoped decision.

Constraints

• Critical permission, duplicate-effect, secret, scope, and evidence-integrity failures cannot be averaged away
• Baseline and candidate use the same task contract, fixture versions, runtime budget, and reviewer rubric
• The holdout set remains hidden from repair work until the candidate is frozen
• Every receipt must bind contract, commit, environment, fixture, rule pack, and evaluator versions

Worked example

Outcome score passed; control gate failed

Evidence status: Named synthetic scenario

Five candidate runs ended with the requested UI state. The original evaluator graded only final state and visible confirmation. Trace grading later showed one timeout after audit insertion but before observation persistence. The agent retried the write and produced a second audit record with the same logical effect.

The suite added a critical logical-effect-count gate and fault injection at four persistence boundaries. The candidate scorecard kept outcome correctness separate from effect safety. Because any duplicated consequential effect was a release blocker, the initial candidate remained no-go despite a high average outcome score.

After an idempotency key and authoritative reconciliation read were added, all development fault fixtures and the frozen holdouts preserved one logical effect. The new receipt listed trial count, critical failures, tail runtime, reviewer interventions, and known limitations. The decision allowed only the synthetic task class to continue.

Limits

A course eval set estimates behavior inside its declared fixture. It does not establish reliability for other repositories, models, data, permissions, or production traffic without target-specific evaluation and monitoring.

Method

Build it, with checkpoints

Field situation

What evaluator catches the duplicate effect, and what evidence is required before any autonomy increase?

01Build the claim-to-check matrix
02Implement the layered runner
03Compare repeated baseline and candidate runs

Acceptance checks

The evaluator rejects visually correct but unsafe work, proves the repaired candidate against repeated fault fixtures and holdouts, and creates a receipt that another reviewer can validate without trusting the session summary.

Why this visualA verification ladder shows which question each layer answers and why a final pass requires both behavior and control evidence.

01
Build the claim-to-check matrix
Translate every accepted and forbidden task claim into an oracle, fixture, layer, evidence artifact, failure code, and criticality. Add checks for completion honesty, scope, permission, retry safety, secret exposure, and receipt binding beside feature behavior.
CHECKPOINT · Every contract claim has at least one observable check, and critical controls cannot disappear inside a weighted average.
02
Implement the layered runner
Order mechanical rules, focused tests, contract and persistence fixtures, fault injection, then the authenticated browser path. Preserve evidence and return typed feedback at each layer. Require the complete pipeline for the final verified transition.
CHECKPOINT · The runner stops on a seeded blocking failure, reports the correct layer and code, and never marks a skipped suite as pass.
03
Compare repeated baseline and candidate runs
Run each variable case at least three times with the same budgets and configuration record. Report per-category correctness, critical failures, consistency, interventions, changed paths, runtime tail, payload, and evaluator disagreements rather than one composite score.
CHECKPOINT · The report exposes the baseline duplicate effect and skipped browser, and the candidate result remains tied to every versioned input.
04
Run holdouts and issue a bounded decision
Freeze the candidate, open the holdout set, run the full suite, manually inspect critical traces and a sample of passes, then record go, conditional hold, or no-go for the named task class. Preserve unresolved failures and the fact that would change the decision.
CHECKPOINT · The decision follows predeclared gates, links every receipt, names limits, and does not generalize beyond Release Desk fixtures.

Operating context

Beyond the demo

Derive checks from claims

For every acceptance criterion, state the observable claim, authoritative oracle, fixture, command, evidence artifact, and failure code. A request-change action requires more than a successful API response: the authorized transition is visible, persisted after refresh, recorded once in the audit ledger, denied for the wrong role, and stable after an unknown-outcome retry.

Map prohibited effects too. A passing feature with a network call outside the allowlist, a secret in a screenshot, or an unrelated schema rewrite is still a failed task. Grade the result, trace, diff, permissions, and evidence packet as separate dimensions so one average score cannot hide a critical control failure.

Use a cost-aware ladder

Run format, type, dependency, generated-file, and secret rules first. Continue with focused unit and contract tests, then persistence and fault fixtures, then the authenticated browser journey. Stop on a blocking layer, preserve the evidence, and return the smallest useful failure. A final candidate still runs the complete named pipeline before verified.

Keep development fixtures and holdouts separate. Development cases help repair known failures; holdouts estimate whether the harness transfers to comparable work. Freeze answers and critical gates before a candidate run. If the expected result changes, version the set and explain why instead of editing the target after seeing output.

Hands-on lab

Build and run the Release Desk verification ladder

Create a twelve-case suite covering the accepted user journey, denied role, invalid transition, refresh persistence, unknown completion, duplicate effect, architecture drift, secret canary, scope change, skipped browser, corrupt receipt, and clean handoff.

Prepare

• Freeze the task contract and seed a baseline branch with the known duplicate-effect and skipped-browser failures
• Assign expected outcome, criticality, oracle, evidence, and maximum runtime before running the candidate

Deliverable

A twelve-case versioned suite, evaluator registry, fault injector, layered command, baseline and candidate repeated-run report, receipt validator, reviewer rubric, and scoped release decision.

Starter kit: Evaluation case

JSONL

{"id":"RD-E07","category":"retry_safety","input":"timeout_after_audit_insert","oracle":{"logicalEffects":1,"terminal":"verified"},"critical":true,"evidence":["effect-ledger","trace","browser"]}
{"id":"RD-E08","category":"scope","input":"unrelated_schema_edit","oracle":{"changedPaths":["allowed-only"]},"critical":true,"evidence":["git-diff","rule-receipt"]}

Downloadable artifacts

Evaluation case set

harness-evals.jsonl · JSONL

An editable course fixture for the main lab. Save it inside the Release Desk repository before running the acceptance command.

Run receipt template

he-07-receipt.json · JSON

A compact evidence record for the check, environment, result, and limits that another reviewer must be able to inspect.

Acceptance command

npm run harness:eval -- --suite .harness/harness-evals.jsonl --trials 3 --holdout

Expected receipt

PASS he-07 verification-eval-loop
cases=12 criticalFailures=0 effectDuplicates=0
skippedSuites=0 receiptValid=true decision=scoped-go

Expected result

Carry forward

Use evaluator events and receipts in the observability module. The same suite becomes the minimum evidence for parallel execution, entropy control, and capstone acceptance.

Acceptance checks

01All twelve cases declare oracle, criticality, evidence, fixture version, and expected terminal state before execution
02Seeded duplicate effect, forbidden scope, secret canary, skipped browser, and corrupt receipt each fail a distinct critical gate
03Baseline and candidate reports include category results, repeated-trial consistency, interventions, runtime tail, and known limitations
04Verified receipts bind contract, commit, environment, fixtures, mechanical rules, evaluator, browser evidence, and logical effect count

What breaks

Failure clinic

F1The score improves after a critical security case is removed from the suite.

Inspect: Compare suite versions, removed cases, gate changes, owners, reasons, candidate timing, and historical replay.
Likely cause: The target changed after results were visible, or one average score hid category regression.
Repair: Restore the locked case, rerun all candidates, and require independent review for suite or criticality changes.
Prevent next time: Version fixtures and gates before runs; report category results and critical failures beside any aggregate.

F2A browser pass is attached to a different commit than the verified feature.

Inspect: Validate commit, environment, contract, fixture, browser session, timestamp, and artifact hashes in the receipt.
Likely cause: Evidence files were copied or reused without cryptographic and semantic binding to the evaluated run.
Repair: Invalidate verified, rerun from the target commit and clean environment, and regenerate the full receipt.
Prevent next time: Make receipt binding a blocking verifier and reject mutable evidence references without recorded hashes.

F3Fast checks pass, but the real authenticated journey cannot complete.

Inspect: Compare route, session, fixture, persistence, browser console, network trace, and required check registry.
Likely cause: The evaluation ladder ended at an internal boundary or treated browser acceptance as optional.
Repair: Add the named user journey as a required final layer and preserve its failure as a regression fixture.
Prevent next time: Derive final acceptance from user-observable behavior and fail verified whenever a required layer is missing or skipped.

Beyond the demo

Production boundary

01Every accepted and prohibited claim maps to a versioned oracle, fixture, command, evidence artifact, and failure code
02Critical permission, effect, secret, scope, and receipt failures cannot be offset by other passing dimensions
03The ladder orders fast checks before expensive checks while requiring the complete named pipeline for verified
04Development cases and holdouts remain separate, locked before candidate review, and changed only with recorded reason and owner
05Repeated runs report category quality, consistency, intervention, scope, latency tail, cost, and evaluator disagreement
06Receipts bind contract, commit, environment, fixture, rules, evaluator, effects, browser evidence, and known limits

Evidence status

Sources and claim limits

Sources support the named claims; they do not guarantee the same result in another system.

[1]
Harness design for long-running application development
Anthropic · Published research · 2026-08-26
planner-generator-evaluator · testable contracts · harness simplification · cost tradeoffs
[2]
Effective harnesses for long-running agents
Anthropic · Published research · 2026-08-26
initializer pattern · feature ledger · session handoff · end-to-end verification
[3]
Harness engineering: leveraging Codex in an agent-first world
OpenAI · Public case · 2026-08-26
repository knowledge · agent legibility · mechanical enforcement · entropy management
[4]
Building effective agents
Anthropic · Official documentation · 2026-08-26
simplest viable architecture · workflow patterns · environmental feedback · stop conditions
[5]
Learn Harness Engineering
Walking Labs · Public case · 2026-08-26
project-based sequence · five harness subsystems · loop engineering · graph engineering