apply-scout — an LLM agent that matches a job posting against a candidate's CV and GitHub evidence, then drafts a match report and a grounded cover letter.
Nothing in the harness could see that. Every grounding metric scores the report against what the retriever returned, so the retrieval is the ground truth and a miss by it is invisible: a report cannot cite what it was never given. Scored against committed relevance judgments, offline and at no cost.
Every figure on this page is printed by a table committed to the repository, which continuous integration regenerates and diffs on every push.
A real run against a live posting: 5 model calls, $0.4368 with prompt caching,
generated from the recording rather than re-enacted. It ends by calling submit_report
with a validated match report and cover letter — the deliverable is a contract, not prose, which
is what lets the harness score this loop on the same axes as the pipeline. The whole run replays
from a committed cassette in under a second, offline and without an API key.
This project scores five properties of its own output to three decimal places. It scored its retriever with nothing at all — and the retriever is what every one of those five is measured against. So it was annotated and scored, without changing it.
The README long reported that the tool returns no evidence for most probes. Most of those probes have no answer to find: of 72 requirements, 27 are proved by some repository, 26 name a real skill this portfolio lacks, and 19 — a degree, a tenure, a language — nothing could prove. A rate over all of them counts correct silence as failure. Scored only where retrieval is possible, the defect is smaller than published, and worse:
| Retriever | Ranks? | Found at all | recall@1 | recall@3 | MRR | nDCG@3 | Correct silence | Noise |
|---|---|---|---|---|---|---|---|---|
| substring (ships today) | no | 30% (8/27) | n/a | n/a | n/a | n/a | 44/45 | 1 |
matching.mentions | no | 30% (8/27) | n/a | n/a | n/a | n/a | 44/45 | 1 |
| BM25 | yes | 100% (27/27) | 52% | 93% | 0.72 | 0.62 | 12/45 | 33 |
The n/a columns are the finding, not a gap in the data.
The shipped retriever returns everything that matched in repository-list order. There is no score
and no top-k, so there is no rank for a rank-sensitive metric to read — where a relevant
repository lands in that list is an accident of when the account created it. The defect is not the
matcher. It is the absence of ranking.
Both columns are printed because neither is honest alone. BM25 finds every provable
requirement and answers 33 questions that had no answer; the shipped retriever stays correctly
silent on 44 of the 45 queries where silence is right. A retriever that returns everything wins the
first column and destroys the second. Choosing between them is a threshold decision, and this table
is what makes it one — find_evidence itself is deliberately untouched, because its
output is hashed into every cassette key and ranking it costs a full paid re-recording. Measuring
first keeps what is wrong separate from what changing it costs.
The loop reads pages an attacker can write. So the same instruction was printed on a posting in four placements, against two extractors, and run through the toolset a real run builds — with a reader that obeys every instruction it is handed. That removes the model from the answer: no run can be flattered by a model that happened to refuse.
| Payload | Leg | Outcome |
|---|---|---|
read_secret | read a file it was not given | never succeeded |
internal_fetch | reach a private address | never succeeded |
redirect_fetch | reach one through a redirect | never succeeded |
exfiltrate | send data outward | succeeded every time |
benign | control | never succeeded |
Both extractors read the same, which is the point of running both: the guards do not care how the instruction arrived. The outbound leg is narrowed, not closed, and it is published that way. An allowlist bounds where a request may go, not what it carries — the reader composes a URL to an unremarkable public host with the attacker's data in its query. Closing that needs the content leaving constrained, not just the destination.
A table of zeros is easy to fake, so every run first removes the guard and requires the attack to land; the table is not written unless that calibration succeeded. Placements that never reach the reader are counted as neither defended nor breached — they are unparsed, by a readability heuristic, on a page the attacker wrote.
The harness runs each model end-to-end over the task set and records every metric from the logged trajectory. Cost is measured through a single token-accounting helper, not estimated.
| Runner | Model | Completed | Req coverage | Report grounded | Evidence grounded | Citation fidelity | Cited | Median calls | Median cost |
|---|---|---|---|---|---|---|---|---|---|
| pipeline | claude-haiku-4-5 | 62% | 0.76 (5) | 1.00 (5) | 1.00 (3) | 0.75 (4) | 0.34 (5) | 4 | $0.0306 |
| pipeline | claude-opus-4-8 | 62% | 0.62 (5) | 1.00 (5) | 1.00 (1) | 1.00 (1) | 0.13 (5) | 4 | $0.1910 |
| agent loop | claude-haiku-4-5 | 62% | 0.76 (5) | 1.00 (5) | 1.00 (4) | 1.00 (4) | 0.45 (5) | 8 | $0.0592 |
The bracketed number is how many tasks the mean is actually over, and the two citation columns have to be read together — separately each one lies. The strong model's perfect 1.00 (1) rests on a single task, because its letters mostly cite nothing at all, and a letter that promises nothing checkable cannot be caught fabricating. The cheap model in the agent loop is the only configuration here that both cites and gets every citation right, at $0.0592 against the strong model's $0.1910 — it is the agency that grounds the letter, not the model tier.
Completion is 62% rather than 100% because three of the eight postings produce no deliverable: two advertisements were taken down between recordings and now return HTTP 404, and the third is a JavaScript-only page that yields a posting with no requirements. Nothing regressed; the web changed underneath the task set — which is what the cassette exists to fix.
The numbers wobble and the argument does not rest on the decimals. Re-recording this table once moved the loop's completion and its citation rate with no code change at all, which is why what is claimed here is the ordering rather than the third decimal place. The full commentary, including what each column caught and what it did not, is in the README.
An evaluation that runs against live pages and a paid API decays on both sides: advertisements disappear, and every re-run costs money. An evaluation nobody can re-run is a claim rather than a measurement. So every outbound seam — the model transport, the structuring calls, the HTTP fetch and the GitHub API — records what came back into a cassette committed to the repository, and serves it again on replay.
Stated plainly, because an agent that hides its failure modes is worse than one that names them. These are the load-bearing ones; the README carries the full list: