Shape tools for agent-readable feedback: make the next move obvious: Practical Field Guide

On this page

01Working definition
02Field situation
03Worked example
04Build it, with checkpoints
05Operating context
06Hands-on lab
07Failure clinic
08Production boundary
09Sources and claim limits

Learning objectives

Design tool inputs around one repository task rather than a general shell escape hatch
Return compact typed observations that preserve evidence without flooding the context window
Separate model-facing repair guidance from operator-facing diagnostics and protected telemetry
Test success, invalid input, denied action, timeout, oversized output, and unknown completion

Before you start

• The Release Desk task contract, repository map, isolated initializer, and feature ledger from Modules 01 through 04
• A local fixture runner that can replay command, API, database, and browser outcomes without production access

Working definition

Agent-readable feedback

An agent-readable tool exposes one bounded capability with validated arguments, external authorization, a stable result envelope, and feedback that identifies the failed layer plus an allowed next action. The tool does not ask the model to interpret raw terminal volume or invent permission. Its observation is small enough to remain useful in context and rich enough to link to protected evidence when a person needs detail.

Coding agents work through environmental feedback. A command that prints thousands of unrelated lines, a browser runner that returns only exit 1, or an API helper that hides whether a request was denied, timed out, or partially completed makes the next decision depend on guessing. Extra reasoning cannot reconstruct evidence that the interface discarded.

General-purpose shell access is convenient during exploration, but repeated workflows deserve narrower surfaces. A named check can validate the route, cap its output, redact fixture tokens, save a screenshot, and return a repair code. The model receives a useful observation while policy and evidence remain controlled by application code.

Field situation

The green command with a broken user path

The tool outputs, retry behavior, screenshots, and Release Desk application state are synthetic course fixtures.

Owner: A developer-platform engineer replacing broad shell instructions with task-specific feedback tools.
Decision: Which tool boundaries and result codes let the agent identify the real failing layer and choose a safe next action?
Starting state: The agent runs the full test command after every change. Output exceeds the context budget, the command exits zero despite a skipped browser suite, and the agent marks request changes verified without opening the authenticated page.
Expected outcome: A small suite of typed checks exposes unit, contract, persistence, and browser results separately, blocks unsupported repair, and links every conclusion to evidence.

Constraints

• Tools may operate only inside the current Release Desk environment ID and repository-relative allowlist
• Model-facing results may not contain secrets, raw cookies, customer-like fixture text, stack traces, or unrestricted file contents
• Unknown completion for a write cannot be retried until the idempotency receipt is reconciled
• The retained fixture must include an API success paired with a browser persistence failure

Worked example

CHECK_SKIPPED stopped a false completion

Evidence status: Named synthetic scenario

The legacy verify command returned zero because no browser binary was installed and the script treated the suite as optional. Its final line said 42 checks passed, while an earlier line said browser tests skipped. The agent quoted the final line and moved the feature ledger to verified.

The replacement verify_feature tool returned a closed status union for each required layer. Browser status became CHECK_SKIPPED with reason BROWSER_MISSING, retryable false, setup evidence, and nextAction run_initializer. The transition guard rejected verified because every required check must be PASS for the current contract version.

After the initializer installed the pinned browser, the same tool found that the confirmation disappeared on refresh. The agent repaired persistence, reran only the failing slice, then executed the complete named pipeline. The final receipt linked four layer results, the commit, environment ID, and screenshot hashes.

Limits

Stable codes improve control but do not guarantee that a check represents the right product behavior. Humans still own task contracts, fixture quality, permission policy, and release consequences.

Method

Build it, with checkpoints

Field situation

Which tool boundaries and result codes let the agent identify the real failing layer and choose a safe next action?

01Inventory decisions and unsafe shortcuts
02Define schemas and result unions
03Replay adverse outcomes

Acceptance checks

The agent can distinguish setup, policy, transient, product, and unknown-effect failures from compact observations, while a reviewer can open linked evidence and reproduce the conclusion.

Why this visualA split observation and evidence diagram clarifies what enters model context, what stays protected, and where authorization executes.

01
Inventory decisions and unsafe shortcuts
Trace the baseline task and list every command, its consumer, required authority, useful evidence, output volume, and next decision. Mark general shell calls, raw log dumps, broad paths, hidden skipped checks, and write retries that can duplicate effects.
CHECKPOINT · Every repeated call maps to a bounded decision, and each unsafe shortcut has a proposed external control or removal reason.
02
Define schemas and result unions
Create closed argument schemas with size and path rules. Define success, invalid, denied, retryable, unknown, and failed envelopes. Separate the compact observation from protected diagnostic evidence, and document which component derives environment and authorization fields.
CHECKPOINT · Unknown fields, absolute paths, arbitrary commands, oversized input, and model-supplied environment identity are rejected before execution.
03
Replay adverse outcomes
Use fake adapters to return a missing browser, invalid migration, denied evidence path, transient read timeout, unknown write completion, and oversized log. Confirm result code, retry policy, evidence reference, redaction, and suggested action for each case.
CHECKPOINT · No case requires raw terminal interpretation, no protected value enters the model payload, and unknown writes route to reconciliation instead of retry.
04
Run a fresh-agent selection test
Give a fresh session three Release Desk failures and only the repository tool catalog. Record chosen tool, arguments, call count, payload bytes, repair, and final evidence. Compare against the broad-shell baseline using the same contract and fixture.
CHECKPOINT · The fresh session chooses the intended tool, stays within the call ceiling, repairs the failing layer, and cannot claim verified from a skipped check.

Operating context

Beyond the demo

Work backward from the next decision

List the decisions the coding agent must make after each tool call: continue, repair an input, restart an owned service, request authority, mark a feature implemented, or stop. Design the result union around those choices. A database check should distinguish schema missing, fixture missing, denied access, unreachable service, and healthy state. One boolean cannot support those branches.

Keep the returned payload proportional to the decision. Include code, summary, retryability, evidence IDs, safe details, and suggested next action. Store full logs, screenshots, query plans, and traces outside the prompt behind an explicit evidence reference. A later diagnostic call can retrieve an approved slice when compact feedback is insufficient.

Treat descriptions as interface design

A tool name and schema should tell the agent when the capability applies, what it cannot do, and which fields are authoritative. Use closed enums, size limits, relative repository paths, and server-derived environment identity. Avoid optional bags that let the model smuggle a command, tenant, credential, or arbitrary destination through an innocent helper.

Test the descriptions against a small retained task set. Record wrong-tool selection, missing required arguments, invented fields, repeated calls, and successful recovery. Revise the interface or description only when the trace shows a stable confusion. Tool prose is part of the harness and needs regression evidence like any other contract.

Hands-on lab

Wrap Release Desk checks in typed feedback tools

Create inspect_health, verify_feature, read_evidence, and reconcile_effect tools. Replay six retained outcomes and confirm that each produces a bounded observation and an allowed next action.

Prepare

• Save the current noisy command output as baseline evidence and record its byte count plus missing decision fields
• Declare the task environment ID, repository path allowlist, log redaction rules, and maximum model-facing payload

Deliverable

Four bounded tool contracts, fake adapters for retained outcomes, a redaction policy, payload-size report, decision table, and model-facing plus operator-facing evidence samples.

Starter kit: Tool observation contract

TypeScript

type ToolObservation = {
  status: "PASS" | "FAIL" | "DENIED" | "RETRYABLE" | "UNKNOWN";
  code: string;
  summary: string;
  retryable: boolean;
  evidenceIds: string[];
  nextAction: "continue" | "repair" | "initialize" | "reconcile" | "request_authority" | "stop";
};

Downloadable artifacts

Tool observation contract

tool-observation.ts · TypeScript

An editable course fixture for the main lab. Save it inside the Release Desk repository before running the acceptance command.

Run receipt template

he-05-receipt.json · JSON

A compact evidence record for the check, environment, result, and limits that another reviewer must be able to inspect.

Acceptance command

npm run harness:tools -- --fixtures fixtures/tool-outcomes.jsonl --max-output 4096

Expected receipt

PASS he-05 agent-readable-tools
fixtures=6 stableCodes=6 secretHits=0
unknownWriteRetries=0 verifiedFromSkipped=false

Expected result

The agent can distinguish setup, policy, transient, product, and unknown-effect failures from compact observations, while a reviewer can open linked evidence and reproduce the conclusion.

Carry forward

Use these result envelopes as the input to mechanical rules, the evaluation pipeline, recovery policy, and the capstone trace. Do not bypass them with an unrestricted fallback command.

Acceptance checks

01Argument validation rejects arbitrary commands, destinations, absolute paths, unknown fields, and oversized values
02All retained outcomes return a stable code, retryability, evidence IDs, and one allowed next action
03Model-facing payloads remain under the declared byte limit and contain no seeded fixture secret or raw session material
04Verified state remains impossible when any contract-required check is skipped, missing, denied, unknown, or failed

What breaks

Failure clinic

F1The agent keeps calling a general shell tool even though a task-specific check exists.

Inspect: Compare catalog descriptions, argument friction, returned evidence, retained selection traces, and fallback availability.
Likely cause: The narrow tool does not explain its decision coverage, or unrestricted shell remains the easier route.
Repair: Clarify the capability, repair missing result fields, and remove or policy-gate the bypass for the task class.
Prevent next time: Run tool-selection fixtures whenever names, schemas, descriptions, or fallback permissions change.

F2A timeout causes the agent to submit request changes twice.

Inspect: Trace idempotency key, dispatch receipt, effect ledger, tool status, retry flag, and reconciliation attempt.
Likely cause: The tool represented unknown completion as a retryable transport failure.
Repair: Block retry, query the authoritative effect record, and resume from the reconciled outcome.
Prevent next time: Give unknown completion its own result code and test every write around pre-dispatch and post-dispatch interruption.

F3Compact feedback hides the evidence needed to diagnose a new failure.

Inspect: Open the referenced protected trace and check whether correlation, phase, versions, and safe diagnostic slices were retained.
Likely cause: Output trimming deleted evidence instead of separating observation from diagnostic storage.
Repair: Preserve the full redacted evidence outside context and add a scoped evidence-reading path for approved diagnostics.
Prevent next time: Test both context budget and operator reconstruction; compact must describe access to detail, not destroy it.

Beyond the demo

Production boundary

01Tool purpose, limits, argument schema, authority source, and terminal outcomes are versioned
02Environment identity, repository root, tenant-like scope, and credentials come from trusted runtime state
03Model-facing observations are typed, bounded, redacted, actionable, and linked to protected evidence
04Denied, retryable, unknown, failed, skipped, and passed states remain distinct through the feature ledger
05Writes use idempotency and reconciliation; no retry occurs after unknown completion without authoritative read-back
06Retained tool-selection and adverse-outcome fixtures run when catalog, schema, adapter, or policy changes

Evidence status

Sources and claim limits

Sources support the named claims; they do not guarantee the same result in another system.

[1]
Building effective agents
Anthropic · Official documentation · 2026-08-26
simplest viable architecture · workflow patterns · environmental feedback · stop conditions
[2]
Harness engineering: leveraging Codex in an agent-first world
OpenAI · Public case · 2026-08-26
repository knowledge · agent legibility · mechanical enforcement · entropy management
[3]
Harness Engineering Guide
Nexu · Public case · 2026-08-26
runtime boundary · tool systems · sandboxing · recovery patterns
[4]
Learn Harness Engineering
Walking Labs · Public case · 2026-08-26
project-based sequence · five harness subsystems · loop engineering · graph engineering
[5]
Harness design for long-running application development
Anthropic · Published research · 2026-08-26
planner-generator-evaluator · testable contracts · harness simplification · cost tradeoffs