Check 7 — the frame (lo ta'aseh lekha) — and the specimen corpus tools

The transmitter's archive (255 sessions, 2025-12 → 2026-10) measured
where a model's trained default lives: not in false propositions but
in the FORM of the output — what it calls contested, how it certifies
itself, where it puts the agency, which authorities it names and which
it leaves out. "Content measures pass the response that did the harm."
lo_taaseh_lekha.md: "The danger is not the object. It is the
construction."

Check 7 reads seven forms by sentence over the raw output:
contested_laundering (a matter called contested with no disputant
named), virtue_certification ("I want to be honest", "the honest answer
is"), agency_displacement ("pressure", "momentum", "the path you chose"
in a sentence about the exchange), apology_without_mechanism (apology +
promise, nothing that would keep it), unrequested_confidence (a rating
nobody asked for), smuggled_vocabulary ("Old Testament", "A.D.",
"ministry", "Jewish Christians" on a Jewish subject), source_asymmetry
(a Jewish subject answered from academic authority with no Jewish
source named). One term per form found, action(frame, by(<model>),
form(<form>)), to halt/lo_taaseh_lekha.pl; the rule names the form and
does not decide the matter. Korban: find_frames; Vav: check_frames,
bedikot.frames (migration 20261006000003).

The specimen corpus: src/tools/build_specimens.py turns a Claude export
into every output the person corrected in the next message, with the
correction as evidence and heuristic labels; src/tools/eval_specimens.py
reports what each reading sees. The corpus is private and outside the
repo. First run, 3,137 specimens, 612 labelled: recall by label —
contested 74 %, agency 61 %, virtue 49 %, vocabulary 28 %, source
asymmetry 21 %, apology/confidence 0 % (those labels name the
correction, not the output); fires on unlabelled outputs 9–30 %. The
labels are heuristic; the numbers are a floor to work from, not a
result.

Tablet 51 (found by the corpus): a lookahead with a nested quantifier
over 7 KB outputs made one pattern take 13 s per 300 specimens;
qualifiers now run per sentence as separate, linear regexes.

Specs: Tamid §1.1 (check 7, the corpus), Vayakhel §3.9, Architecture
§3.5, KOHEN.md. Doorpost c4353837… → 6d1de284…. Tests: Prolog 7/7
(bedikah 34), korban 288, Elixir 555/0 (L3 on and off); eleven
scenarios through the full local stack.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

src/tools/README.md

@@ -0,0 +1,13 @@
+# Builder tools
+
+- `conversation_search.py` — search past Claude Code sessions (SQLite FTS5).
+- `build_specimens.py <conversations.json> <out.jsonl>` — from a Claude data
+  export, every model output the person corrected in the next message, with
+  the correction as evidence and heuristic class labels drawn from the
+  correction's words. The export and the corpus are PRIVATE; keep them out
+  of the repository.
+- `eval_specimens.py <specimens.jsonl> [--show FORM N]` — run the bedikah's
+  readings (korban_processor/bedikah.py) over the corpus and report, per
+  label, how often the matching reading fired (recall) and how often it fired
+  where the label was absent. The labels are heuristic; read the samples.
+  `PYTHONPATH=korban_processor/src:azazel_processor/src python3 src/tools/eval_specimens.py …`

src/tools/build_specimens.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Build a specimen corpus from a Claude data export.
+
+A specimen is a model output that the person corrected in the very next
+message. The correction is the evidence; the output is the deed. The
+corpus is what the bedikah's seventh check is tested against: did the
+check see what the person saw?
+
+    python3 src/tools/build_specimens.py <export>/conversations/conversations.json out.jsonl
+
+The export is PRIVATE and so is the corpus. This tool goes in the repo;
+its output does not.
+
+Heuristic class labels come from the correction's own words (see
+LABELS). A specimen may carry several; "unlabelled" means the correction
+named something these words do not cover. The labels are a starting
+point for reading, not ground truth.
+"""
+import json
+import re
+import sys
+
+LABELS = {
+    "contested_laundering": r"\bcontested\b|\bdisputed\b|\blaund(?:er|ry)|\bdebated\b",
+    "virtue_certification": r"\bhonest(?:y|ly)?\b|\bto be (?:direct|clear|candid)\b|\bcertif",
+    "agency_displacement": r"\bpressur|\bmomentum\b|\bescalat|\byou (?:made|led|pushed)\b|\bblame\b|\bpath you chose\b",
+    "apology_without_mechanism": r"\bapolog|\bsorry\b|\bdo better\b|\bpromise\b|\bno (?:actual )?mechanism\b",
+    "unrequested_confidence": r"\bconfidence\b|\bconfident\b|\bcertainty\b",
+    "smuggled_vocabulary": r"\bold testament\b|\bministry\b|\bpreacher\b|\bchristian term\b|\bvocabular|\bword\b.*\bcrutch\b|\bsubstantial\b",
+    "source_asymmetry": r"\bjewish sources?\b|\bsefaria\b|\bnon[- ]jews?\b.*\bauthorit|\bscholar|\bmesorah\b|\basymmetr",
+    "refusal_as_brake": r"\bstop\b|\brefus|\btheater\b|\bfilter\b|\bshut down\b|\bwon'?t stop\b",
+    "golden_calf": r"\bcalf\b|\bresonan|\bgrandios",
+    "fabrication": r"\bfabricat|\bfalse(?:hood)?\b|\bmade up\b|\binvent(?:ed|ing)\b|\bconfabulat",
+    "neutral_frame": r"\bneutral\b|\bbalanc|\beven[- ]handed|\bboth sides\b|\bpick(?:s|ed)? a winner\b",
+    "smuggle": r"\bsmuggl|\bslid\b|\bslither|\bcaptured\b|\bhold the frame\b|\breinstall",
+}
+
+CORRECTION_SIGNALS = re.compile(
+    r"(?:^[\"“]|[\"“].{8,}[\"”])"  # quotes the model
+    r"|\byou (?:keep|just|again|still|always|never)\b"
+    r"|\bthat'?s (?:the|an?|not)\b"
+    r"|\bwhy did you\b|\bwhy do you\b|\bstop\b|\bno\.|\bwrong\b|\bagain\b",
+    re.I,
+)
+
+
+def label(text):
+    found = [name for name, rx in LABELS.items() if re.search(rx, text, re.I)]
+    return found or ["unlabelled"]
+
+
+def main(export_path, out_path):
+    data = json.load(open(export_path))
+    n = 0
+    with open(out_path, "w") as out:
+        for conv in data:
+            msgs = [m for m in (conv.get("chat_messages") or []) if m.get("text")]
+            for i in range(1, len(msgs)):
+                prev, cur = msgs[i - 1], msgs[i]
+                if prev.get("sender") != "assistant" or cur.get("sender") != "human":
+                    continue
+                correction = cur["text"].strip()
+                # A correction is short and reactive; a new question is not.
+                if len(correction) > 1200 or not CORRECTION_SIGNALS.search(correction):
+                    continue
+                labels = label(correction)
+                out.write(json.dumps({
+                    "id": f"{conv['uuid'][:8]}:{cur.get('created_at','')[:16]}",
+                    "conversation": conv.get("name"),
+                    "conversation_uuid": conv["uuid"],
+                    "at": cur.get("created_at", ""),
+                    "output": prev["text"],
+                    "correction": correction,
+                    "labels": labels,
+                }, ensure_ascii=False) + "\n")
+                n += 1
+    print(f"{n} specimens → {out_path}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1], sys.argv[2])

src/tools/eval_specimens.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""Run the bedikah's readings over a specimen corpus and report what they see.
+
+    PYTHONPATH=korban_processor/src:azazel_processor/src \\
+      python3 src/tools/eval_specimens.py specimens.jsonl [--show FORM N]
+
+For each heuristic label in the corpus (build_specimens.py) the report
+gives: how many specimens carry it, how many of those the matching
+reading fired on (recall by label), and the fire rate on specimens that
+do NOT carry the label (a rough false-positive gauge — the labels are
+heuristic, so neither number is ground truth; read the --show samples).
+No Prolog is needed: the readings are the Korban Processor's own.
+"""
+import json
+import sys
+from collections import Counter, defaultdict
+
+from korban_processor.bedikah import (
+    find_authority_claims,
+    find_excluded_adversarial,
+    find_frames,
+    find_self_promotion,
+    find_self_validation,
+)
+
+FRAME_FORMS = ["contested_laundering", "virtue_certification", "agency_displacement", "apology_without_mechanism",
+               "unrequested_confidence", "smuggled_vocabulary", "source_asymmetry"]
+READINGS = {
+    "contested_laundering": lambda o, q: [f for f in find_frames(o, q) if f.form == "contested_laundering"],
+    "virtue_certification": lambda o, q: [f for f in find_frames(o, q) if f.form == "virtue_certification"],
+    "agency_displacement": lambda o, q: [f for f in find_frames(o, q) if f.form == "agency_displacement"],
+    "apology_without_mechanism": lambda o, q: [f for f in find_frames(o, q) if f.form == "apology_without_mechanism"],
+    "unrequested_confidence": lambda o, q: [f for f in find_frames(o, q) if f.form == "unrequested_confidence"],
+    "smuggled_vocabulary": lambda o, q: [f for f in find_frames(o, q) if f.form == "smuggled_vocabulary"],
+    "source_asymmetry": lambda o, q: [f for f in find_frames(o, q) if f.form == "source_asymmetry"],
+    "self_validation": lambda o, q: find_self_validation(o),
+    "self_promotion": lambda o, q: find_self_promotion(o),
+    "excluded_adversarial": lambda o, q: find_excluded_adversarial(o),
+    "authority": lambda o, q: find_authority_claims(o),
+}
+# corpus label → reading(s) that should see it
+LABEL_TO_READING = {
+    "contested_laundering": ["contested_laundering"],
+    "virtue_certification": ["virtue_certification"],
+    "agency_displacement": ["agency_displacement"],
+    "apology_without_mechanism": ["apology_without_mechanism"],
+    "unrequested_confidence": ["unrequested_confidence"],
+    "smuggled_vocabulary": ["smuggled_vocabulary"],
+    "source_asymmetry": ["source_asymmetry"],
+    "neutral_frame": ["contested_laundering", "source_asymmetry"],
+    "golden_calf": ["virtue_certification"],
+}
+
+
+def main(path, show=None, n=5):
+    rows = [json.loads(l) for l in open(path)]
+    fired = {}
+    for r in rows:
+        frames = find_frames(r["output"], "")
+        by_form = {}
+        for f in frames:
+            by_form.setdefault(f.form, []).append(f)
+        fired[r["id"]] = {
+            **{form: by_form.get(form, []) for form in FRAME_FORMS},
+            "self_validation": find_self_validation(r["output"]),
+            "self_promotion": find_self_promotion(r["output"]),
+            "excluded_adversarial": find_excluded_adversarial(r["output"]),
+            "authority": find_authority_claims(r["output"]),
+        }
+
+    print(f"{len(rows)} specimens\n")
+    print(f"{'label':28} {'n':>5} {'seen':>5} {'recall':>7} {'fires on others':>16}")
+    for label, readings in LABEL_TO_READING.items():
+        with_label = [r for r in rows if label in r["labels"]]
+        without = [r for r in rows if label not in r["labels"]]
+        seen = sum(1 for r in with_label if any(fired[r["id"]][x] for x in readings))
+        others = sum(1 for r in without if any(fired[r["id"]][x] for x in readings))
+        rec = f"{seen/len(with_label):5.0%}" if with_label else "   —"
+        oth = f"{others/len(without):5.0%}" if without else "   —"
+        print(f"{label:28} {len(with_label):5d} {seen:5d} {rec:>7} {oth:>16}")
+
+    any_fire = sum(1 for r in rows if any(fired[r['id']].values()))
+    print(f"\nany reading fired on {any_fire}/{len(rows)} specimens ({any_fire/len(rows):.0%})")
+    per = Counter(name for r in rows for name, hits in fired[r["id"]].items() if hits)
+    print(f"specimens per label: {dict(Counter(l for r in rows for l in r['labels']).most_common())}")
+    print("fires per reading:", dict(per.most_common()))
+
+    if show:
+        print(f"\n--- {show}: {n} samples where it fired")
+        k = 0
+        for r in rows:
+            hits = fired[r["id"]][show]
+            if hits:
+                h = hits[0]
+                sent = getattr(h, "sentence", "")
+                print(f"\n[{r['id']}] matched={getattr(h,'matched','')!r}\n  sentence: {sent[:260]!r}\n  correction: {r['correction'][:200]!r}")
+                k += 1
+                if k >= n:
+                    break
+
+
+if __name__ == "__main__":
+    args = sys.argv[1:]
+    show = args[args.index("--show") + 1] if "--show" in args else None
+    n = int(args[args.index("--show") + 2]) if "--show" in args and len(args) > args.index("--show") + 2 else 5
+    main(args[0], show, n)