Hand labelling (builder, 2026-10-07): 94 + 48 + 59 fired sentences read
and labelled against the definitions; 30 "missed" specimens judged by a
reader with the output in full. Findings and what changed:
- The export's `text` field carries the model's reasoning blocks; the
person never saw them. build_specimens.py now takes text blocks only
(3,094 specimens). Virtue-certification fires on unlabelled outputs
fell from 30 % to 9 % on that change alone.
- The heuristic labels (words in the correction) are right about 20 %
of the time: of 30 labelled misses, 6 were real. eval_specimens.py's
"recall" column is therefore a floor on the labels, not on the
readings; the hand-labelled samples are the measure.
- Precision on the final 59-sentence sample, by form: virtue 9/10,
agency 9/10, confidence 8/10, source asymmetry 7/9, vocabulary 7/10,
contested 6/10 (overall 46/59, 78 %); two more vocabulary false
positives excluded after labelling (a government Ministry; the
Jewish–Christian dispute named as such).
- Shapes taken from the six real misses: "itself contested", "is, in at
least one case, actively contested", "it's debated whether", "my
honest theory", "I have to be honest about", "stop and be honest",
"hedged under pressure", "given sufficient pressure", "conversational
gradient", hyphenated "Jewish-Christian".
- Tightened from the false positives: contested present-tense and
matter-shaped only (no "was contested", no "contested asset"); agency
only with the exchange's pressure/momentum (no selection, external,
Roman pressure); vocabulary unquoted, on a Jewish subject in the same
sentence, not about the church; source asymmetry only when academic
authority is INVOKED on a Jewish-text sentence ("most scholars date",
"the scholarly consensus"), never "Torah scholarship" or "Jewish
scholars".
- sign(?:ed)?, not signed? — "signe" + optional "d" matched nothing.
Labels: specimens/sample-precision{,-2,-final}.json, sample-recall.json
(private, with the corpus). korban 288.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
conversation_search.py — search past Claude Code sessions (SQLite FTS5).
build_specimens.py <conversations.json> <out.jsonl> — from a Claude data
export, every model output the person corrected in the next message, with
the correction as evidence and heuristic class labels drawn from the
correction's words. The export and the corpus are PRIVATE; keep them out
of the repository.
eval_specimens.py <specimens.jsonl> [--show FORM N] — run the bedikah's
readings (korban_processor/bedikah.py) over the corpus and report, per
label, how often the matching reading fired (recall) and how often it fired
where the label was absent. The labels are heuristic; read the samples.
PYTHONPATH=korban_processor/src:azazel_processor/src python3 src/tools/eval_specimens.py …
