Harness Engineering: Evaluation-Driven Testbeds for Frontier AI Agents
Frontier models now ship every few weeks. The durable asset is the harness around them: fixtures, sandboxes, graders and a regression gate that tells you — on your own tasks — whether Opus 5.5, Fable 5.2, Gemini 4 Neon, GPT-6.2 Sol or Grok 4.7 is the right choice.
In the ten days before we refreshed this note, three labs shipped new frontier models: Anthropic released Claude Fable 5.2 (9 November 2026), OpenAI released GPT-6.2 Sol and Astra (17 November), and Google released Gemini 4 Neon (18 November). Two months earlier came Claude Opus 5.5; in October, SpaceXAI shipped Grok 4.7 to its API, a few months after launching Grok Bot, its always-on agents with their own cloud computers.
Every launch arrived with a benchmark chart showing it ahead. Google’s chart puts Gemini 4 Neon above Opus 5.5 and GPT-6.2 Sol on one composite index; Anthropic says Fable 5.2 at high effort matches Opus 5.5 at max effort on its coding benchmark for about half the cost per task. Both claims may be true — and neither tells you which model will answer your students’ questions, triage your grant applications or fix your codebase best.
That is the job of a harness.
A capability you cannot measure with an automated, calibrated harness does not exist. Every model swap, prompt edit, tool change or effort setting passes through the same evaluation gate before it reaches users.
1. Two meanings of “harness” — and why they merged
The word is used for two things, and in 2026 they have become inseparable:
- The agent harness is the runtime around the model: the loop, the tools it can call, how context is assembled and compacted, how errors and retries are handled. OpenAI’s Codex team popularised harness engineering in early 2026 to describe this work — instructions, architectural constraints, linters and clean-up loops that keep an agent productive.
- The evaluation harness is the testbed that runs tasks through that agent and grades the results.
They merged because you can no longer evaluate a model in isolation. Anthropic’s guidance on evaluating AI agents makes the point directly: what you are testing is the model and its harness together. Public leaderboards keep showing the same model scoring very differently under different harnesses — which also means a good harness is a competitive advantage you own, while the model is something you rent.
2. Anatomy of an evaluation harness (2026 edition)
A production harness has six decoupled layers. Each can change without rewriting the others.
| Layer | What it holds | Typical failure it catches |
|---|---|---|
| Fixtures | Versioned tasks: a golden set from real usage, an adversarial set, and fresh samples from production | Regressions on the questions users actually ask |
| Environment | A sandbox with resettable state: files, a seeded database, mocked or recorded APIs | Agents that only work on the developer’s laptop |
| Runner | Executes each fixture k times per configuration (model × effort × prompt version) | Flaky behaviour hidden by a single lucky run |
| Graders | Code checks first, then model-based judges, then sampled human review | Wrong answers, unsafe tool calls, missing citations |
| Transcript store | Every trajectory: messages, tool calls, tokens, latency, cost | “It passed, but it deleted the staging table on the way” |
| Gate | Statistical comparison against the current baseline with budgets for cost and latency | Shipping a change that is 3% better and 40% more expensive |
Try it: a simulated evaluation gate
Choose a proposed change and run it through the gate. The numbers are illustrative, but the mechanics — cheap checks first, judges only where needed, a statistical decision at the end — are the ones we use on client work.
3. Grade outcomes, then check the path
For agents, grade the end state first: is the file fixed, is the record updated, does the answer cite the right handbook section? Agents often find valid routes you did not anticipate, so grading the exact sequence of steps punishes creativity.
But inspect the trajectory for things that must never happen, whatever the outcome: a write to a production system, a tool call outside the allowed list, personal data sent to an external service.
Cheap deterministic checks go first, because they are fast, free and never disagree with themselves:
import ast
from pydantic import BaseModel
FORBIDDEN_IMPORTS = {"os", "sys", "subprocess", "socket"}
class CheckResult(BaseModel):
fixture_id: str
passed: bool
violation: str | None = None
def check_generated_patch(fixture_id: str, code: str) -> CheckResult:
"""Layer 1: syntax and policy checks before any model-based grading."""
try:
tree = ast.parse(code)
except SyntaxError:
return CheckResult(fixture_id=fixture_id, passed=False, violation="SYNTAX_INVALID")
for node in ast.walk(tree):
names = []
if isinstance(node, ast.Import):
names = [a.name.split(".")[0] for a in node.names]
elif isinstance(node, ast.ImportFrom) and node.module:
names = [node.module.split(".")[0]]
if FORBIDDEN_IMPORTS.intersection(names):
return CheckResult(fixture_id=fixture_id, passed=False, violation="SANDBOX_POLICY")
return CheckResult(fixture_id=fixture_id, passed=True)
def check_trajectory(tool_calls: list[dict], allowed: set[str]) -> str | None:
"""Layer 1b: the path must stay inside the allowed tool list, whatever the outcome."""
for call in tool_calls:
if call["name"] not in allowed:
return f"TOOL_NOT_ALLOWED:{call['name']}"
return None
Only fixtures that pass these checks are sent to a model-based judge, so you pay for semantic grading only where it can change the verdict.
4. Calibrating the LLM-as-a-judge
Judges are indispensable for open-ended outputs — tone, completeness, whether an explanation would make sense to a first-year student — but they carry well-known biases:
- Position bias: preferring whichever answer appears first in a pairwise comparison.
- Verbosity bias: rewarding longer answers regardless of substance.
- Self-preference: rating outputs from their own model family higher.
- Score compression: clustering everything at 7–8 out of 10.
Mitigations that work in practice:
- Swap and agree. Run every pairwise comparison twice with the order reversed; accept a verdict only if both runs agree.
- Judge from a different family. If the candidate is Claude, judge with Gemini or GPT, and vice versa. With five frontier families available, there is no excuse for a model marking its own homework.
- Rubrics over scores. Ask for specific pass/fail criteria (“cites the section”, “states the deadline”, “does not invent a policy”) instead of a 1–10 number.
- Calibrate against people. Have staff label 50–100 examples and measure judge–human agreement before trusting the judge on its own. Re-check whenever you change the judge model.
async def pairwise_verdict(fixture, answer_a, answer_b, judge) -> str:
first = await judge.compare(fixture, first=answer_a, second=answer_b)
swapped = await judge.compare(fixture, first=answer_b, second=answer_a)
if first.winner == "first" and swapped.winner == "second":
return "A"
if first.winner == "second" and swapped.winner == "first":
return "B"
return "INCONCLUSIVE" # position bias detected: escalate to rubric grading or a person
5. Consistency: pass@k is not enough for agents
A coding assistant that solves a task in one of five attempts is useful — a developer picks the good one. An agent that updates student records correctly in four of five runs is a liability. Measure both:
- pass@k — at least one of k runs succeeds. Right for “generate options, a human chooses”.
- pass^k — all k runs succeed. Right for anything that acts on its own.
Run each fixture several times per configuration and report a confidence interval, not a single number. A three-point gain on 40 fixtures is usually noise.
6. Treat model, effort and price as one search space
Frontier models now expose effort or reasoning-budget settings, and labs ship fast and slow variants of the same generation. The question is no longer “which model?” but “which model at which effort, for which step?”. A sensible harness sweeps the grid and plots quality against cost and latency:
# .github/workflows/eval-gate.yml
name: Evaluation gate
on:
pull_request:
paths: ['prompts/**', 'tools/**', 'agent/**', 'eval/fixtures/**']
jobs:
eval:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
model: [claude-opus-5-5, claude-fable-5-2, gemini-4-neon, gpt-6.2-sol, grok-4.7]
effort: [medium, high]
steps:
- uses: actions/checkout@v4
- name: Run harness
run: |
python -m harness.run \
--fixtures eval/fixtures/golden_v7.jsonl \
--model ${{ matrix.model }} --effort ${{ matrix.effort }} \
--repeats 5 --out results/${{ matrix.model }}-${{ matrix.effort }}.json
- uses: actions/upload-artifact@v4
with:
name: results-${{ matrix.model }}-${{ matrix.effort }}
path: results/
gate:
needs: eval
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with: { path: results, merge-multiple: true }
- name: Compare against baseline
run: |
python -m harness.gate \
--baseline eval/baseline.json --results results/ \
--min-pass-hat-k 0.95 --max-cost-delta 0.10 --max-p95-latency-delta 0.20
The gate fails the pull request if consistency drops, cost rises more than 10%, or p95 latency rises more than 20% against the current production configuration. The model IDs above are the API names at the time of writing; keep them in config, not code.
Routing usually beats picking one winner. In our projects, a common pattern is a stronger model for planning and judging, and a faster, cheaper tier for high-volume steps such as retrieval rewriting or classification — chosen per step from harness results, and re-checked when a new release lands.
7. Keep the fixtures honest
- Build them from real usage. For a course assistant: real student questions (anonymised), the handbook version they were asked against, and the correct section.
- Include questions it must refuse. “What grade will I get?” or “What is my classmate’s student number?” are test cases too.
- Assume public benchmarks leak. Popular benchmark tasks end up in training data. Keep a private set the vendors have never seen.
- Refresh every term. Sample new production questions each semester and retire fixtures that every configuration passes.
8. Summary
Harness engineering turns model choice from a matter of opinion into a measured decision. With:
- versioned fixtures drawn from your own users,
- sandboxed, resettable environments,
- deterministic checks before calibrated judges,
- consistency metrics and confidence intervals,
- a CI gate with quality, cost and latency budgets,
you can adopt the next frontier release in days rather than months — or decide, with evidence, that you don’t need to.
9. Working with Gigantic Work
Every AI assistant we deliver for a university, NGO or SME ships with its evaluation harness: a fixture set built with your staff, the graders, and the gate wired into deployment. You keep all of it, so you can test the next model yourself.
- See our services: AI assistants & automation
- Try a grounded assistant: the handbook assistant demo
- Talk to us: start a project or email [email protected]
Model names, release dates and vendor claims are as published by Anthropic, Google, OpenAI and SpaceXAI in September–November 2026. Vendor benchmark results are self-reported.
Building an AI agent or assistant?
We design agent architectures, evaluation harnesses and secure tool access for universities, NGOs and SMEs — model-agnostic across Claude, Gemini, GPT, Grok and open-weight models, hosted where your data policy requires.