---
name: human-review-packet
description: Build the packet a human needs to validate machine-generated rows — every claim shipped next to the source URL and the local snapshot of the text it came from, so the reviewer checks the work instead of redoing it. Use this skill WHENEVER you are about to ask a person to label, vet, spot-check, hand-check, verify, QA, review, approve, sign off on, or mark hit/miss on rows, records, extractions, classifications, labels, enrichment output, scraped data, or model output — including when you are only about to say "can you check these" or "does this look right". Use it BEFORE writing the review CSV, sheet, or markdown, not after, because the packet's shape determines whether the review happens at all: a file where the reviewer must go and find the source themselves is a file that gets deferred session after session. Use it even when the check sounds trivial ("just eyeball these ten"), because the recurring failures here are invisible in the artifact — a value with no evidence behind it looks identical to a supported one, a shifted column silently attaches a label to the wrong person, and free-text where an enum was needed produces answers that cannot be joined back to anything.
---

# Human review packet — ship the evidence, not just the claim

An agent asking a human to validate rows is asking for the most expensive
attention in the pipeline. The reviewer is slow, finite, easily discouraged, and
the only thing standing between a confident extraction and a database full of
plausible fiction.

The failure is almost never that the human refuses. It is that the artifact makes
reviewing cost nearly as much as doing the work again — so it gets postponed,
and the postponement is invisible because nothing errors.

**The rule this skill exists to enforce:** never ask a human to check a claim
without putting the evidence for that claim next to it. Not a list of pages the
agent read. Not "source: their website". The specific URL, and a local snapshot
of the text that was actually read at extraction time.

## Onboarding — start here

### 1. What this skill does

Turns "can you check these rows?" into a packet the reviewer can work through
without opening a browser, running a search, or asking you what a column means.
It covers what goes in the artifact, how to sample, how to avoid anchoring the
reviewer to the model's answer, and what to do with the marks when they come back.

It applies to any human-validation step: ICP extractions, headline
classifications, magnet detection, enrichment output, employer verification,
scraped-field spot checks.

### 2. What it can't do — say this before promising anything

| The user wants | Reality |
| --- | --- |
| A review packet from data with no provenance | It cannot invent provenance. If the fetch step never stored the URL and the raw text, the packet cannot carry them — fix the fetch step and re-run, or say plainly the review is unverifiable. |
| To skip snapshots because "the URL is enough" | A live page changes. When it does, the reviewer sees text that no longer contains the quote and marks a correct extraction as a miss. Snapshot or don't ask. |
| The agent to do the validation itself | Then it is not human validation, and the number that comes out cannot be quoted as one. Route to an eval or an LLM-judge and label it as such. |
| A statistically valid accuracy figure from a spot check | A 10-row hand-check is a gate, not a measurement. Do not report it as accuracy. See `references/sampling.md`. |
| To re-draw the sample after seeing the first one | That converts a check into a search for a clean sample. The draw is fixed once, with its seed recorded. |
| A packet for data the reviewer has already seen labelled | Anchoring. If the point is measuring agreement, prior/model answers must be hidden — see the blind/reviewed split below. |

Reach for `gsheets-api` when the packet should land in a spreadsheet with
enforced dropdowns rather than a CSV.

### 3. Setup

Reads whatever the extraction step already wrote, so the real requirement is
upstream: **the fetch or scrape step must persist, per record, the source URL and
the raw text it read.** If that is missing, that is the bug — fix it there.

For delivery it needs `token.json` (gitignored) for the Sheets API. Never hand
over a CSV and ask someone to import it — see "Delivery" below.

```bash
python scripts/push_to_sheets.py --csv data/packet.csv --title "Review" \
    --checkbox checked --checkbox supported --out data/packet-sheet.json
```

**Verify** the data can support a packet at all, before promising one:

```bash
python -c "
import json,sys
recs=json.load(open(sys.argv[1]))
n=len(recs)
u=sum(1 for r in recs if r.get('source_urls'))
print(f'{n} records, {u} with source_urls  ->', 'OK' if u==n else 'PROVENANCE GAP')
" data/icp-batch3.json
```

### 4. How to invoke

It should fire on its own from the description. Ask for it directly with
"build the review packet" or "use human-review-packet".

### 5. Cost

Free. No API calls, no credits. The cost it manages is the reviewer's attention,
which is the scarcest input in the pipeline and the only one with no rate card.

### 6. The rest of the skill

- `references/failure-modes.md` — eight review failures observed in this repo,
  each with what it looked like at the time and the guard that prevents it
- `references/packet-spec.md` — the required columns, the snapshot convention,
  and the blind/reviewed split
- `references/sampling.md` — fixed draws, what a spot check can and cannot claim

---

## The procedure

### Step 1 — Check provenance exists before promising a packet

Every claim needs three things: the **value**, the **verbatim span** that supports
it, and the **URL that span is on**. If any is missing, stop and say so. A packet
built on two of three teaches the reviewer to trust unverifiable rows.

A populated value beside an empty evidence span is not a minor gap. It is the
model asserting something the source never said, and in the artifact it looks
exactly like a supported value. **Sort those to the top and mark them
explicitly** — they are the highest-yield rows in the review.

### Step 2 — Resolve each span to the exact page it is on

Not the list of pages read. If a record cites three URLs, "it came from one of
these" hands the search back to the reviewer, which is the thing that makes
review feel like redoing the work.

Spans are verbatim by construction, so the page is findable:

- Collapse whitespace on both sides before matching — extracted spans and cached
  markdown differ in whitespace often enough to break a literal find.
- **Split multi-span values on their joiner** (commonly `" / "`) and match the
  longest single span. The joined string was never on any page, so searching for
  it finds nothing and reports a false "unlocatable".
- A span that genuinely matches no cached page is a real finding. Surface it as
  such rather than leaving the cell blank.

### Step 3 — Ship the snapshot, not just the link

Include a path to the locally cached text that was read at extraction time.

The live page is not the evidence. It changes, and when it does the reviewer
marks correct extractions as misses and loses confidence in the whole set. The
snapshot is what the claim was actually made against.

Convention already used in this repo: `data/icp-pages/<sha1(url)[:12]>.md`.
Keep the keying stable so any packet can resolve a URL to its snapshot.

### Step 4 — Put the columns the reviewer types into first

`mark` and `note` are column A and B. Everything else is context.

This sounds cosmetic. It is not — the review CSV this replaced had the mark
column past 60 columns of context, and the check was deferred four sessions.

### Step 5 — Constrain the answer space

Free text where an enum was needed produces answers that cannot be joined back
to anything. Give the reviewer the closed vocabulary, in the packet, and enforce
it with dropdowns when the packet is a sheet.

Watch for the reviewer answering a *different question* than the column asks —
axis-1 reasoning appearing in an axis-2 column is the tell. Say in the header
exactly what is being judged and what is explicitly not.

### Step 6 — Decide blind vs reviewed, and never mix them

- **Blind** — no model answer shown. Measures the human against the source.
- **Reviewed** — model answer shown, human marks hit/pass. Faster, but anchored.

Both are legitimate; they measure different things. Mixing them in one file
produces a number belonging to neither. If a person is being re-reviewed on rows
they labelled before, their prior answers stay hidden.

### Step 7 — Deliver a link, never a file to import

**A CSV handed to a human is not a delivered packet.** Importing it into Sheets
needs an explicit File > Import > Replace current sheet; a plain Drive upload
lands as a blob that will not open as a spreadsheet at all. Both failures are
silent, and the reviewer concludes the data "didn't show up" — which is the point
at which the review stops.

Push it through the Sheets API and hand back a URL that opens into the work:

```bash
python scripts/push_to_sheets.py --csv data/packet.csv \
    --title "ICP hand-check 11" --checkbox checked --checkbox supported \
    --dropdown verdict=hit,miss --out data/packet-sheet.json
```

That script applies the things this skill requires and a CSV cannot carry:
checkboxes on answer columns, **strict** dropdowns on enum columns (strict, not
warn — a warning still lets an out-of-vocabulary answer through), a frozen
header, a wrapped quote column, and column widths that keep the answer cells on
screen.

**Answer columns come in pairs.** One checkbox cannot distinguish "this is wrong"
from "I haven't reached this row yet", and those two produce very different
loads. Use `checked` (I have looked at this) alongside `supported` (the evidence
holds). A dropdown is the alternative when there are more than two outcomes.

Persist the spreadsheet id and url next to the CSV so the next revision can
update **that** sheet rather than creating a second one — see
`versioned-artifacts` for what happens when it doesn't.

### Step 8 — Explain the task in the conversation. The sheet holds rows only.

**The instructions go where the reviewer already is, which is the chat — not in
the spreadsheet.** The spreadsheet is the place they score; it is not a document.
Splitting attention across both means one of them gets skipped, and putting prose
in a sheet makes the sheet feel like reading rather than ticking.

So: write the explanation as a message, and hand over a link to a sheet that
contains data and nothing else. No `START HERE` tab, no instruction rows above the
header, no notes column pre-filled with guidance.

(`push_to_sheets.py --readme` exists for the case where the reviewer is **not** in
the conversation — an outside labeller, someone picking it up next week. That is
the exception. Default to the chat.)

The explanation in chat must contain, in this order:

1. **The one question**, stated in a single sentence, in the reviewer's language —
   not the schema's. "Does that sentence actually prove that claim?" beats "verify
   enum-quote correspondence".
2. **What each answer column means**, including why there are two boxes rather
   than one.
3. **Two worked examples, from the reviewer's own data** — one obvious pass, one
   real failure, both copied from actual rows. Invented examples are ignored;
   examples the reviewer recognises are read. The failure example should be the
   *characteristic* mistake, because it teaches the pattern rather than the rule.
4. **What they can skip**, with a count. Rows already decided (unquoted values,
   pre-filled notes) should be named and dismissed — "14 of these are automatic,
   that leaves 35 to think about" turns an intimidating grid into a short job.
5. **How to check the source themselves**, and when to use the snapshot instead of
   the live URL.
6. **What happens with the marks**, and what a *clustered* set of misses means.
7. **A time estimate.**

Write it at the reading level of someone who did not build the pipeline. No field
names in the prose where a plain phrase will do, no reference to the schema, no
"simply" or "just".

Hide the columns the reviewer does not need (`--hide`). Context they cannot act on
still competes for attention. It stays in the sheet; it just is not on screen.

```bash
python scripts/push_to_sheets.py --csv data/packet.csv --title "…" \
    --checkbox checked --checkbox supported --link source_url \
    --readme data/packet-readme.txt \
    --hide cached_page --hide confidence
```

### Step 9 — State the return path in the artifact itself

The last line of the packet says what happens with the marks: *"tell me the
misses and I load all 45"*. A review with no stated consequence is a review that
feels optional.

### Step 8 — Report what the check can support

A ten-row gate answers "is this generation safe to load", not "what is the error
rate". Report it as the former. If the misses cluster in one field, that field is
the finding — fix the extractor for it rather than hand-correcting rows.