An LLM agent that matches a job posting against a candidate's CV and GitHub evidence,
then drafts a match report and a grounded cover letter — a tool loop written from scratch, with
safety budgets and a trajectory-evaluation harness.
One run, as it happened
A real run against a live posting on 2026-08-21: 5 model calls, 33 evidence probes,
$0.4368 with prompt caching ($0.5195 without), 125 seconds. Generated from the recording rather than re-enacted. It ends by calling
submit_report with a validated match report and cover letter — the deliverable is a
contract, not prose, which is what lets the harness score this loop on the same axes as the
pipeline. The whole run replays from a committed cassette in under a second, offline and without an
API key.
Evaluation: when is the cheaper model enough?
The harness runs each model end-to-end over the task set and records every metric from the
logged trajectory. Cost is measured through a single token-accounting helper, not estimated.
| Runner | Model | Completed | Req coverage | Report grounded | Evidence grounded | Citation fidelity | Cited | Median calls | Median cost |
|---|
| pipeline | claude-haiku-4-5 | 75% | 0.76 (5) | 1.00 (5) | 1.00 (3) | 0.75 (4) | 0.29 | 4 | $0.0291 |
| pipeline | claude-opus-4-8 | 75% | 0.62 (5) | 1.00 (5) | 1.00 (1) | 1.00 (1) | 0.11 | 4 | $0.1827 |
| agent loop | claude-haiku-4-5 | 62% | 0.76 (5) | 1.00 (5) | 1.00 (4) | 1.00 (4) | 0.45 | 8 | $0.0592 |
The loop row is recorded with prompt caching and the pipeline rows are not: on the
same basis its median task costs $0.0741. It is also a fresh sample — re-recording it moved
completion from 75% to 62% and the citation rate from 0.74 to 0.45 with no code change. Eight tasks
is a small sample and it wobbles; what survived both runs is the ordering, not the decimals.
Report grounded is the newest column and the honest reading of it is
that it caught nothing. It scores how much of a report traces back to the posting that was actually
fetched — the blind spot the citation columns cannot see, because a letter can cite its report
perfectly while the report itself was invented, which is what an earlier recording of the
JavaScript-only task did: ten requirements lifted from the candidate's own CV, every citation valid.
A report rating requirements with no posting behind it scores 0.00 rather than "not applicable", so
that run would have shown. It no longer exists in the cassette — re-recording the loop produced a
reply in which the model refuses to assess instead of inventing — so the column is a control that
fired nowhere, pinned by unit tests rather than by the task set.
Evidence grounded closes the last link of the chain: tool retrieves
evidence → report cites it → letter cites the report. The citation columns score only the
last hop, so a fabricated link that reaches the report becomes a valid citation target and launders
itself into a perfect fidelity. This one scores the report's links against what
github_evidence actually returned, compared at repository level rather than by URL
string — the tool returns a README's link while a report may cite the repository root, and
treating those as two sources produced a false positive the first time this was measured by hand.
Scored properly it reads 1.00: every cited link in the recording traces to a repository the tools
retrieved.
The bracketed number is how many tasks the mean is actually over: a task with no
annotation has no coverage to measure, and a letter that cites nothing has no fidelity. Read the two
citation columns together — separately each one lies. The strong model's perfect 1.00 rests on a
single task, because in five of six its letters cite nothing at all (11% of sentences carry a link),
and a letter that promises nothing checkable cannot be caught fabricating.
The third row is the from-scratch tool loop solving the same tasks on the same axes. It
costs about 2.5× the pipeline on the same model and basis, and it is the only configuration here
that both cites (45% of sentences) and gets every citation right. Grounded, substantiated letters
therefore cost $0.0592 with the cheap model in the loop against $0.1827 for the strong model in the
pipeline — a third of the price, better coverage, four times the citation rate. It is the agency that
grounds the letter, not the model tier.
Prompt caching, measured inside each recorded run by pricing the same prompts as if
nothing had been cached: the 38-turn eval loop saved 26% ($0.3994 → $0.2950) and
the 5-turn demo 16% ($0.5195 → $0.4368). Per task the saving ran from 36% down to
nothing — the Reddit task saved 0%, because the loop gave up after one call and nothing was ever
re-sent. Caching pays for turns over a growing prefix.
On this task set the strong model costs roughly six times more per task and does not
buy a better match report: the same completion and lower requirement coverage. It does not buy a
better letter either — that took a metric fix to see, since its headline fidelity came from letters
that cite almost nothing. What produces grounded letters here is giving the cheap model the agent
loop, at roughly half the strong model's price.
Requirement coverage is the share of human-annotated skills the extraction found —
recall only, deliberately. The annotation lists the handful of skills a human judged load-bearing,
not all two dozen requirements in a posting, so a precision denominator would penalise reading the
posting thoroughly. This column used to report an exact-match F1 of 0.33 and 0.23; on this task set
a flawless extraction could not have scored above 0.68 under that metric. Re-scoring cost nothing,
because the metric is a pure function of the recorded runs below.
Completion is 75% rather than 100% because two of the eight advertisements were taken
down between the first run and this one and now return HTTP 404. Nothing in the pipeline regressed;
the web changed underneath the task set — which is what the cassette below exists to fix.
Reproducibility: record once, replay forever
An evaluation that runs against live pages and a paid API decays on both sides: advertisements
disappear, and every re-run costs money. An evaluation nobody can re-run is a claim rather than a
measurement. So every outbound seam — the model transport, the structuring calls, the HTTP fetch and
the GitHub API — records what came back into a cassette committed to the repository, and serves it
again on replay. Main-text extraction is recorded as well, although it never leaves the machine: the
extraction library's output shifts between its own versions, so a replay that re-ran it would derive
different text from the same recorded page and miss every entry keyed on it.
- The table above reproduces offline with
--cassette-mode replay: no network, no API key, no cost. Continuous integration
replays it on every pull request and every push to the main branch, which turns the published
numbers into a regression test.
- Replay never falls back to the network. An unrecorded request stops the run;
quietly serving it live would turn a reproducible evaluation back into a paid, unverifiable one.
- Cost survives the offline path. Tokens and dollars are replayed as captured at
recording time, so the cost column stays a measurement rather than a zero.
- A prompt edit invalidates exactly what it touches — the cassette key hashes the
whole request, system prompt included — so "change a prompt, re-run the harness" is enforced by the
machinery instead of by memory.
Recording the whole eight-posting, two-model table cost $0.88 and produced 68 entries.
Every reproduction since has been free.