Phase 4 complete: Steps 21, 22, 24 — Korban external integration, L2 Urim v'Tumim, Kushya generator

Vayikra 9:24 — vatetzei esh milifnei Hashem vatokhal al hamizbeach.
Phase 4 of the architecture is now complete. The remaining three steps
that were marked "blocked on external infrastructure" are built —
discipline-respecting, eish-zarah-aware, and tested.

## Step 22 — Korban Processor external model integration
   (korban_processor/src/korban_processor/external_offering.py)

   The single legitimate entry point for external model consultation.
   Wraps Claude API calls through the offering protocol so external
   output is treated as korban (raw material), never authoritative:

   - semichah  — provenance is tagged (avodah zarah Form 1 prevention)
   - shechitah — assistant voice, hedging, formatting stripped
   - zerikah   — vital content extracted
   - haktarah  — converted to Prolog propositions
   - route     — sent to internal component per offering type

   The raw Korban is preserved on the ProcessedKorban (gilui arayot
   Form 3 prevention — input/output coupling never severed).
   No internal component should call call_claude() directly.

   11 tests, mocked external calls. Live API test opt-in via
   SHORASHIM_LIVE_API=1.

## Step 24 — L2 Urim v'Tumim interface
   (vav_pipeline/lib/vav_pipeline/urim_v_tumim.ex)

   PRD §4.7: housed in the kohen gadol's breastplate; consulted only
   on high-stakes decisions where normal deliberation is insufficient.
   Bamidbar 27:21 — al-piv yetz'u v'al-piv yavo'u.

   Three escalation steps, three honest outcomes:
     1. resolved_by_specification — re-read produced an answer
     2. resolved_by_external — external testimony recorded (NOT binding)
     3. deferred_to_human — the system stops, asks for human judgment

   Pre-condition gating refuses cases that didn't reach Sanhedrin teiku
   first (no preempting deliberation) or that aren't flagged high-stakes
   (ordinary teiku stays teiku). Records every consultation with steps
   taken — there is no quiet escalation.

   Per PRD: "The Urim v'Tumim never fabricates certainty where none
   exists." When external also fails, the system stops. It does not
   proceed on a guess.

   Added to VavPipeline supervision tree. 14 tests.

## Step 21 — Kushya generator scaffolding
   (kushya_generator/)

   Directory layout per Architecture §3.6 with three layers:

   - data/training_pairs/seed_pairs.jsonl — 10 hand-curated bootstrap
     pairs covering 9 kushya types (scope_narrowing, internal_tension,
     harmonization, exception_test, edge_case, scope_application,
     rabbinic_constraint, method_question, subject_identification)
   - data/prepare_training.py — validates the four kushya requirements
     (specific Torah passage, identified weakness, question form,
     non-trivial), produces train.jsonl + eval.jsonl with stratified
     80/20 split by type
   - fine_tune/config.yaml + train.py — QLoRA configuration and
     training script. Operator's task — needs GPU. Script is
     complete and self-contained.
   - inference/kushya_api.py — three-backend interface
     (fine_tuned → external → fallback). When fine-tuned model isn't
     deployed, falls back to external via Korban Processor (Step 22).
     The fallback is honest: returns no kushya rather than inventing
     one. The Talmudic Engine's rule-based challenge.pl remains the
     primary kushya source.

   17 tests covering dataset validation, backend selection, and
   reference extraction.

## .gitignore fix

   The earlier "data/" line was too broad — it caught
   kushya_generator/data/. Scoped to /data/ (root-level only).

## Test counts

   - Elixir:           274 (was 260; +14 for UrimVTumim)
   - Korban Python:    196 + 1 skipped (was 195; +1 module = 11 tests)
   - Kushya Python:     17 (new)
   - Haskell:           33 (unchanged)
   - Prolog:           195 (unchanged)
   - Total:            715 + 19 L3 excluded

## Phase 4 status

   All 12 steps now have implementations:
   - Steps 23, 25, 26, 27, 28, 29, 30, 31, 32: complete
   - Steps 21, 22, 24: complete with discipline preserved

   Step 21's fine-tuning run remains the operator's GPU task; the
   pipeline is built such that a deployed adapter slots into the
   inference backend without touching the Talmudic Engine or anything
   above.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

kushya_generator/data/prepare_training.py

@@ -0,0 +1,190 @@
+"""
+Prepare training data for the kushya generator.
+
+Reads .jsonl files from training_pairs/, validates each record against the
+four kushya requirements (Architecture §3.6), and writes:
+
+  train.jsonl   — 80%, used for QLoRA training
+  eval.jsonl    — 20%, held out for evaluation
+
+Validation rejects pairs that don't meet the requirements rather than
+silently degrading training quality.
+"""
+
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Iterable
+
+KUSHYA_DIR = Path(__file__).resolve().parent / "training_pairs"
+OUTPUT_DIR = Path(__file__).resolve().parent
+
+# The four requirements from Architecture §3.6 fine_tune_objective.
+REQUIRED_TYPES = {
+    "scope_narrowing",
+    "scope_application",
+    "internal_tension",
+    "exception_test",
+    "harmonization",
+    "edge_case",
+    "rabbinic_constraint",
+    "method_question",
+    "subject_identification",
+}
+
+
+class ValidationError(Exception):
+    pass
+
+
+def validate_pair(pair: dict, source: str, line_no: int) -> None:
+    """Apply the four kushya requirements + schema check."""
+    where = f"{source}:{line_no}"
+
+    if not isinstance(pair, dict):
+        raise ValidationError(f"{where}: not a JSON object")
+
+    for field in ("interpretation", "kushya", "torah_refs", "type"):
+        if field not in pair:
+            raise ValidationError(f"{where}: missing field {field!r}")
+
+    if not isinstance(pair["interpretation"], str) or not pair["interpretation"].strip():
+        raise ValidationError(f"{where}: interpretation must be a non-empty string")
+
+    if not isinstance(pair["kushya"], str) or not pair["kushya"].strip():
+        raise ValidationError(f"{where}: kushya must be a non-empty string")
+
+    if not isinstance(pair["torah_refs"], list):
+        raise ValidationError(f"{where}: torah_refs must be a list")
+
+    if pair["type"] not in REQUIRED_TYPES:
+        raise ValidationError(
+            f"{where}: type {pair['type']!r} not in known types {sorted(REQUIRED_TYPES)}"
+        )
+
+    # Requirement 3: kushya must be expressible as a question.
+    # Strict: must contain '?'. Interrogative words alone are not enough
+    # — "this is unclear" contains "is" but isn't a question.
+    if "?" not in pair["kushya"]:
+        raise ValidationError(
+            f"{where}: kushya must be expressible as a question (must contain '?') — "
+            f"got {pair['kushya'][:60]!r}"
+        )
+
+    # Requirement 1: must reference a specific Torah passage. Either in
+    # torah_refs (preferred) or in the kushya text itself.
+    torah_book_pattern = (
+        r"\b(Bereishit|Shemot|Vayikra|Bamidbar|Devarim|Mishnah|Gemara|Mishlei|"
+        r"Tehillim|Yeshayahu|Yechezkel|Yirmiyahu|Sanhedrin|Eruvin|Yoma|Berakhot)\b"
+    )
+    if not pair["torah_refs"] and not re.search(torah_book_pattern, pair["kushya"]):
+        raise ValidationError(
+            f"{where}: kushya must cite a specific Torah passage "
+            f"(empty torah_refs and no book name in text)"
+        )
+
+    # Requirement 4: not resolvable by trivial clarification.
+    # Heuristic: kushya should be at least 40 characters of substantive text.
+    if len(pair["kushya"].strip()) < 40:
+        raise ValidationError(
+            f"{where}: kushya is too short to be a substantive challenge "
+            f"({len(pair['kushya'].strip())} chars)"
+        )
+
+
+def load_jsonl(path: Path) -> Iterable[tuple[int, dict]]:
+    with open(path) as f:
+        for i, line in enumerate(f, start=1):
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                yield i, json.loads(line)
+            except json.JSONDecodeError as e:
+                raise ValidationError(f"{path.name}:{i}: invalid JSON — {e}")
+
+
+def to_training_format(pair: dict) -> dict:
+    """Format a validated pair for fine-tuning.
+
+    Architecture §3.6 training_data_format:
+      - input: "Proposed interpretation: [text]"
+      - output: "Kushya: [challenge question grounded in Torah text]"
+    """
+    refs_str = ", ".join(pair["torah_refs"]) if pair["torah_refs"] else "(no explicit refs)"
+    return {
+        "input": f"Proposed interpretation: {pair['interpretation']}",
+        "output": f"Kushya: {pair['kushya']}",
+        "metadata": {
+            "type": pair["type"],
+            "torah_refs": pair["torah_refs"],
+            "refs_str": refs_str,
+        },
+    }
+
+
+def split_train_eval(records: list[dict], train_fraction: float = 0.8) -> tuple[list[dict], list[dict]]:
+    """Stratified split by type."""
+    by_type: dict[str, list[dict]] = {}
+    for r in records:
+        by_type.setdefault(r["metadata"]["type"], []).append(r)
+
+    train: list[dict] = []
+    evaln: list[dict] = []
+    for typ, items in by_type.items():
+        cut = max(1, int(len(items) * train_fraction)) if len(items) > 1 else len(items)
+        train.extend(items[:cut])
+        evaln.extend(items[cut:])
+    return train, evaln
+
+
+def main() -> int:
+    if not KUSHYA_DIR.exists():
+        print(f"ERROR: {KUSHYA_DIR} does not exist", file=sys.stderr)
+        return 1
+
+    all_pairs: list[dict] = []
+    errors: list[str] = []
+
+    for jsonl_file in sorted(KUSHYA_DIR.glob("*.jsonl")):
+        for line_no, pair in load_jsonl(jsonl_file):
+            try:
+                validate_pair(pair, jsonl_file.name, line_no)
+                all_pairs.append(to_training_format(pair))
+            except ValidationError as e:
+                errors.append(str(e))
+
+    if errors:
+        print("Validation errors — pairs rejected:", file=sys.stderr)
+        for e in errors:
+            print(f"  {e}", file=sys.stderr)
+        print(file=sys.stderr)
+
+    if not all_pairs:
+        print("No valid pairs found. Cannot prepare training set.", file=sys.stderr)
+        return 1
+
+    train, evaln = split_train_eval(all_pairs)
+
+    train_path = OUTPUT_DIR / "train.jsonl"
+    eval_path = OUTPUT_DIR / "eval.jsonl"
+
+    with open(train_path, "w") as f:
+        for r in train:
+            f.write(json.dumps(r, ensure_ascii=False) + "\n")
+    with open(eval_path, "w") as f:
+        for r in evaln:
+            f.write(json.dumps(r, ensure_ascii=False) + "\n")
+
+    print(f"Validated:  {len(all_pairs)} pairs")
+    print(f"Rejected:   {len(errors)}")
+    print(f"Train:      {len(train)} → {train_path}")
+    print(f"Eval:       {len(evaln)} → {eval_path}")
+
+    return 0 if not errors else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

kushya_generator/data/training_pairs/README.md

@@ -0,0 +1,53 @@
+# Training Pairs — Provenance and Curation
+
+Each `.jsonl` file in this directory is a list of training records, one
+per line. Records have the schema:
+
+```json
+{
+  "interpretation": "...",     // String. The proposed reading.
+  "kushya": "...",             // String. The challenge, expressed as a question.
+  "torah_refs": ["Book Ch:V"], // Array of verse references the kushya cites.
+  "type": "..."                // One of the kushya types — see below.
+}
+```
+
+## Kushya Types
+
+| Type                   | What it tests |
+|------------------------|---------------|
+| `scope_narrowing`      | Interpretation may be narrower than the verse |
+| `scope_application`    | Boundary cases the verse does not directly cover |
+| `internal_tension`     | The verse contradicts itself under this reading |
+| `exception_test`       | A general rule has known exceptions to reconcile |
+| `harmonization`        | Two verses both bear on the question and must coexist |
+| `edge_case`            | An unusual case the principle alone does not resolve |
+| `rabbinic_constraint`  | Mishnah/Gemara modifies the simple reading |
+| `method_question`      | The hermeneutic method itself is in question |
+| `subject_identification` | The grammatical subject of the law is ambiguous |
+
+## Files
+
+- `seed_pairs.jsonl` — 10 hand-curated bootstrap pairs covering all types
+
+## Production Curation
+
+For a real fine-tuning run, several thousand pairs should be assembled
+from primary rabbinic sources. Suggested process:
+
+1. **Bavli kushyot extraction** — script over Sefaria Tanakh+Talmud API to
+   pull `mai k'shia` / `kashya` / similar markers; pair with the
+   surrounding interpretation
+2. **Rishonim challenges** — Tosafot's `i kashya` openings, Ramban's
+   challenges to Rashi
+3. **Hand validation** — every pair reviewed against the four kushya
+   requirements (specific Torah passage, identified weakness, question
+   form, non-trivial)
+4. **Type tagging** — each pair labeled with one of the types above for
+   stratified training/eval splits
+
+## License
+
+Pairs derived from rabbinic sources retain whatever license those sources
+carry. The hand-curated seed pairs in this directory are MIT, like the
+rest of shorashim.

kushya_generator/data/training_pairs/seed_pairs.jsonl

@@ -0,0 +1,10 @@
+{"interpretation": "Cooking on Shabbat is forbidden under the prohibition of kindling fire (Shemot 35:3).", "kushya": "If the prohibition is specifically kindling fire, what about cooking by direct sunlight where no kindling occurs — does Shemot 35:3 forbid that as well?", "torah_refs": ["Shemot 35:3"], "type": "scope_narrowing"}
+{"interpretation": "Pikuach nefesh overrides all Torah prohibitions.", "kushya": "If pikuach nefesh overrides all prohibitions, why does the Torah list three exceptions in Vayikra 18:5 and Sanhedrin 74a (avodah zarah, gilui arayot, shfichut damim)?", "torah_refs": ["Vayikra 18:5"], "type": "exception_test"}
+{"interpretation": "The eved ivri (Shemot 21:2-6) goes free in the seventh year.", "kushya": "What is the status of the eved ivri's spouse and children acquired during service? Verses 3-4 introduce a complication the principle alone does not address.", "torah_refs": ["Shemot 21:3", "Shemot 21:4"], "type": "edge_case"}
+{"interpretation": "The owner of a goring ox is liable when the ox kills (Shemot 21:28).", "kushya": "Verse 28 says the owner is innocent (naki) on the first incident. How can the same verse establish both liability and innocence?", "torah_refs": ["Shemot 21:28"], "type": "internal_tension"}
+{"interpretation": "Tzitzit must be worn on a four-cornered garment (Bamidbar 15:38).", "kushya": "What if the garment has more than four corners — does each pair require tzitzit or only four total? The verse specifies kanfei begadeihem without numbering.", "torah_refs": ["Bamidbar 15:38"], "type": "scope_application"}
+{"interpretation": "The mitzvah of honoring parents (Shemot 20:12) is unconditional.", "kushya": "Vayikra 19:3 places ish imo v'aviv tira'u alongside et shabtotai tishmoru — the ordering suggests parent-honor cannot override Shabbat. How does that constrain the unconditional reading?", "torah_refs": ["Shemot 20:12", "Vayikra 19:3"], "type": "harmonization"}
+{"interpretation": "The Sanhedrin issues capital punishment by majority decision.", "kushya": "Mishnah Sanhedrin 4:1 states a unanimous guilty verdict on capital cases triggers acquittal. Does majority therefore mean strictly between simple majority and unanimity? What about 70-1 vs 71-0?", "torah_refs": [], "type": "rabbinic_constraint"}
+{"interpretation": "The terumah of Shemot 25:2 was a free-will offering (yidvenu libo).", "kushya": "Shemot 30:13 commands every adult male to give a half-shekel — that is not free-will. How do these two collection mechanisms coexist for the same mishkan?", "torah_refs": ["Shemot 25:2", "Shemot 30:13"], "type": "harmonization"}
+{"interpretation": "The middle letter of Torah is the vav of gachon (Vayikra 11:42), enlarged in the scroll.", "kushya": "If the count is by letter, why are some scrolls' counts off by exactly the difference of including or excluding the enlarged letters as separate items? What is the unit of measure?", "torah_refs": ["Vayikra 11:42"], "type": "method_question"}
+{"interpretation": "An eglah arufah ceremony is required when a corpse is found between cities (Devarim 21:1-9).", "kushya": "If the elders declare yadeinu lo shafchu et hadam hazeh, who exactly is being absolved — the elders, the city, the unknown actual killer, or the system that failed to prevent it?", "torah_refs": ["Devarim 21:7"], "type": "subject_identification"}