Phase 4 complete: Steps 21, 22, 24 — Korban external integration, L2 Urim v'Tumim, Kushya generator

Vayikra 9:24 — vatetzei esh milifnei Hashem vatokhal al hamizbeach.
Phase 4 of the architecture is now complete. The remaining three steps
that were marked "blocked on external infrastructure" are built —
discipline-respecting, eish-zarah-aware, and tested.

## Step 22 — Korban Processor external model integration
   (korban_processor/src/korban_processor/external_offering.py)

   The single legitimate entry point for external model consultation.
   Wraps Claude API calls through the offering protocol so external
   output is treated as korban (raw material), never authoritative:

   - semichah  — provenance is tagged (avodah zarah Form 1 prevention)
   - shechitah — assistant voice, hedging, formatting stripped
   - zerikah   — vital content extracted
   - haktarah  — converted to Prolog propositions
   - route     — sent to internal component per offering type

   The raw Korban is preserved on the ProcessedKorban (gilui arayot
   Form 3 prevention — input/output coupling never severed).
   No internal component should call call_claude() directly.

   11 tests, mocked external calls. Live API test opt-in via
   SHORASHIM_LIVE_API=1.

## Step 24 — L2 Urim v'Tumim interface
   (vav_pipeline/lib/vav_pipeline/urim_v_tumim.ex)

   PRD §4.7: housed in the kohen gadol's breastplate; consulted only
   on high-stakes decisions where normal deliberation is insufficient.
   Bamidbar 27:21 — al-piv yetz'u v'al-piv yavo'u.

   Three escalation steps, three honest outcomes:
     1. resolved_by_specification — re-read produced an answer
     2. resolved_by_external — external testimony recorded (NOT binding)
     3. deferred_to_human — the system stops, asks for human judgment

   Pre-condition gating refuses cases that didn't reach Sanhedrin teiku
   first (no preempting deliberation) or that aren't flagged high-stakes
   (ordinary teiku stays teiku). Records every consultation with steps
   taken — there is no quiet escalation.

   Per PRD: "The Urim v'Tumim never fabricates certainty where none
   exists." When external also fails, the system stops. It does not
   proceed on a guess.

   Added to VavPipeline supervision tree. 14 tests.

## Step 21 — Kushya generator scaffolding
   (kushya_generator/)

   Directory layout per Architecture §3.6 with three layers:

   - data/training_pairs/seed_pairs.jsonl — 10 hand-curated bootstrap
     pairs covering 9 kushya types (scope_narrowing, internal_tension,
     harmonization, exception_test, edge_case, scope_application,
     rabbinic_constraint, method_question, subject_identification)
   - data/prepare_training.py — validates the four kushya requirements
     (specific Torah passage, identified weakness, question form,
     non-trivial), produces train.jsonl + eval.jsonl with stratified
     80/20 split by type
   - fine_tune/config.yaml + train.py — QLoRA configuration and
     training script. Operator's task — needs GPU. Script is
     complete and self-contained.
   - inference/kushya_api.py — three-backend interface
     (fine_tuned → external → fallback). When fine-tuned model isn't
     deployed, falls back to external via Korban Processor (Step 22).
     The fallback is honest: returns no kushya rather than inventing
     one. The Talmudic Engine's rule-based challenge.pl remains the
     primary kushya source.

   17 tests covering dataset validation, backend selection, and
   reference extraction.

## .gitignore fix

   The earlier "data/" line was too broad — it caught
   kushya_generator/data/. Scoped to /data/ (root-level only).

## Test counts

   - Elixir:           274 (was 260; +14 for UrimVTumim)
   - Korban Python:    196 + 1 skipped (was 195; +1 module = 11 tests)
   - Kushya Python:     17 (new)
   - Haskell:           33 (unchanged)
   - Prolog:           195 (unchanged)
   - Total:            715 + 19 L3 excluded

## Phase 4 status

   All 12 steps now have implementations:
   - Steps 23, 25, 26, 27, 28, 29, 30, 31, 32: complete
   - Steps 21, 22, 24: complete with discipline preserved

   Step 21's fine-tuning run remains the operator's GPU task; the
   pipeline is built such that a deployed adapter slots into the
   inference backend without touching the Talmudic Engine or anything
   above.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

kushya_generator/test/test_dataset.py

@@ -0,0 +1,147 @@
+"""
+Validate the kushya training dataset structure.
+
+Architecture §3.6 — every training pair must satisfy the four requirements
+or be rejected. These tests run prepare_training.py's validator against
+the seed pairs to catch dataset regressions.
+"""
+
+import json
+import sys
+from pathlib import Path
+
+# Add parent data/ dir to path so we can import prepare_training
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "data"))
+
+from prepare_training import (  # noqa: E402
+    REQUIRED_TYPES,
+    ValidationError,
+    load_jsonl,
+    to_training_format,
+    validate_pair,
+)
+
+
+SEED_PATH = (
+    Path(__file__).resolve().parent.parent / "data" / "training_pairs" / "seed_pairs.jsonl"
+)
+
+
+def test_seed_pairs_file_exists():
+    """The seed pairs file must exist — it bootstraps the training pipeline."""
+    assert SEED_PATH.exists(), f"seed_pairs.jsonl not found at {SEED_PATH}"
+
+
+def test_seed_pairs_validate():
+    """Every seed pair must pass validation. If a pair fails, the dataset
+    has regressed and the training pipeline would silently degrade."""
+    failures = []
+    for line_no, pair in load_jsonl(SEED_PATH):
+        try:
+            validate_pair(pair, "seed_pairs.jsonl", line_no)
+        except ValidationError as e:
+            failures.append(str(e))
+    assert not failures, "seed_pairs.jsonl has invalid records:\n" + "\n".join(failures)
+
+
+def test_seed_pairs_cover_all_kushya_types():
+    """The seed set should exercise every kushya type, so the training
+    split is stratified across the categories."""
+    types_seen = set()
+    for _, pair in load_jsonl(SEED_PATH):
+        types_seen.add(pair["type"])
+
+    # We don't require ALL types to be in the small seed (10 pairs),
+    # but we do require at least 3 types to ensure stratification works.
+    assert len(types_seen) >= 3, (
+        f"Only {len(types_seen)} types in seed: {types_seen}. "
+        f"Need at least 3 for stratified split."
+    )
+
+    # Every type used must be a known type (validator already checks this,
+    # but we want a clear failure if a typo creeps in).
+    unknown = types_seen - REQUIRED_TYPES
+    assert not unknown, f"Unknown types in seed: {unknown}"
+
+
+def test_to_training_format_produces_expected_shape():
+    """The training format must follow Architecture §3.6:
+      input:  'Proposed interpretation: ...'
+      output: 'Kushya: ...'
+    """
+    sample = {
+        "interpretation": "Cooking on Shabbat is forbidden.",
+        "kushya": "What about cooking by direct sunlight where no kindling occurs in Shemot 35:3?",
+        "torah_refs": ["Shemot 35:3"],
+        "type": "scope_narrowing",
+    }
+    formatted = to_training_format(sample)
+    assert formatted["input"].startswith("Proposed interpretation:")
+    assert formatted["output"].startswith("Kushya:")
+    assert formatted["metadata"]["type"] == "scope_narrowing"
+    assert formatted["metadata"]["torah_refs"] == ["Shemot 35:3"]
+
+
+# ============================================================
+# Validator regression tests — these should REJECT bad pairs.
+# ============================================================
+
+def _expect_rejection(pair, must_contain):
+    try:
+        validate_pair(pair, "test", 1)
+    except ValidationError as e:
+        assert must_contain in str(e), (
+            f"Expected error containing {must_contain!r}, got: {e}"
+        )
+    else:
+        raise AssertionError(f"Pair should have been rejected: {pair}")
+
+
+def test_validator_rejects_empty_kushya():
+    pair = {
+        "interpretation": "Some interp.",
+        "kushya": "",
+        "torah_refs": ["Shemot 21:1"],
+        "type": "internal_tension",
+    }
+    _expect_rejection(pair, "kushya must be a non-empty string")
+
+
+def test_validator_rejects_kushya_without_question_form():
+    pair = {
+        "interpretation": "Some interp.",
+        "kushya": "There is a problem with this interpretation that needs clarification.",
+        "torah_refs": ["Shemot 21:1"],
+        "type": "internal_tension",
+    }
+    _expect_rejection(pair, "expressible as a question")
+
+
+def test_validator_rejects_too_short_kushya():
+    pair = {
+        "interpretation": "Some interp.",
+        "kushya": "Why?",
+        "torah_refs": ["Shemot 21:1"],
+        "type": "internal_tension",
+    }
+    _expect_rejection(pair, "too short")
+
+
+def test_validator_rejects_no_torah_reference():
+    pair = {
+        "interpretation": "Some interp.",
+        "kushya": "What does this even mean in context — there is internal tension here is there not?",
+        "torah_refs": [],
+        "type": "internal_tension",
+    }
+    _expect_rejection(pair, "must cite a specific Torah passage")
+
+
+def test_validator_rejects_unknown_type():
+    pair = {
+        "interpretation": "Some interp.",
+        "kushya": "What does this mean in Shemot 21:1, given the apparent tension?",
+        "torah_refs": ["Shemot 21:1"],
+        "type": "made_up_type",
+    }
+    _expect_rejection(pair, "not in known types")

kushya_generator/test/test_kushya_api.py

@@ -0,0 +1,124 @@
+"""
+Tests for the kushya generator API.
+
+These tests use mocked external callers; no real API requests are made.
+"""
+
+import sys
+from pathlib import Path
+from unittest.mock import patch
+
+# Add parent inference/ dir
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "inference"))
+# Add the shorashim repo's korban_processor src to sys.path
+sys.path.insert(
+    0,
+    str(Path(__file__).resolve().parent.parent.parent / "korban_processor" / "src"),
+)
+
+from kushya_api import KushyaResult, generate_kushya, _extract_refs  # noqa: E402
+
+
+# ============================================================
+# Backend selection
+# ============================================================
+
+def test_fallback_returns_no_kushya_with_explanation():
+    """The fallback path honestly reports no challenge — never invents one."""
+    result = generate_kushya("Some interpretation.", backend="fallback")
+    assert isinstance(result, KushyaResult)
+    assert result.kushya is None
+    assert result.backend == "fallback"
+    assert "does not invent" in (result.notes or "")
+
+
+def test_fine_tuned_without_endpoint_falls_back_in_auto():
+    """auto mode skips fine_tuned when no endpoint is configured, then
+    skips external when not available, ending in fallback."""
+    # We expect the Korban Processor's external model call to either succeed
+    # (if API key is present) or fail (if not) — either way, the result has
+    # a clear backend. Mock to ensure deterministic behavior.
+    from korban_processor.external_offering import consult_external_model  # noqa: F401
+
+    with patch("kushya_api.consult_external_model") as mock_consult:
+        mock_consult.side_effect = RuntimeError("simulated API failure")
+        result = generate_kushya("Some interpretation.", backend="auto")
+
+    assert result.kushya is None
+    assert result.backend in ("fallback", "external")
+
+
+def test_fine_tuned_explicit_without_endpoint_raises():
+    """When backend='fine_tuned' is forced but no endpoint is configured,
+    we raise rather than silently falling through."""
+    import pytest
+    with pytest.raises(ValueError, match="fine_tuned_endpoint"):
+        generate_kushya("Some interpretation.", backend="fine_tuned")
+
+
+# ============================================================
+# External backend — mocked, exercises the full eish-zarah path
+# ============================================================
+
+def test_external_backend_returns_kushya_via_korban_processor():
+    """When external consultation succeeds, the result is recorded with
+    'external' backend and a notes field that emphasizes it is testimony,
+    not ruling."""
+    from kushya_api import _try_external
+
+    fake_consultation = type("FakeConsultation", (), {})()
+    fake_consultation.raw_response = type(
+        "FakeResponse", (), {"content": "raw response with a question?"}
+    )()
+    fake_consultation.processed = type(
+        "FakeProcessed",
+        (),
+        {"vital_content": "What about Shemot 35:3 — does the prohibition extend to passive heating?"},
+    )()
+
+    with patch("kushya_api.consult_external_model", return_value=fake_consultation):
+        result = _try_external("Cooking on Shabbat is forbidden.", allow_fallback=True)
+
+    assert result.kushya
+    assert result.backend == "external"
+    assert "testimony" in (result.notes or "").lower()
+    assert "Shemot 35:3" in result.torah_refs
+
+
+def test_external_backend_handles_empty_vital_content():
+    """If zerikah extracts nothing vital from external response, we
+    record None — we do not pretend an empty kushya is real."""
+    from kushya_api import _try_external
+
+    fake_consultation = type("FakeConsultation", (), {})()
+    fake_consultation.raw_response = type("FakeResp", (), {"content": "Great question!"})()
+    fake_consultation.processed = type("FakeProc", (), {"vital_content": ""})()
+
+    with patch("kushya_api.consult_external_model", return_value=fake_consultation):
+        result = _try_external("Some interp.", allow_fallback=True)
+
+    assert result.kushya is None
+    assert result.backend == "external"
+    assert "no vital content" in (result.notes or "").lower()
+
+
+# ============================================================
+# Reference extraction
+# ============================================================
+
+def test_extract_refs_finds_torah_citations():
+    text = "What about Shemot 35:3 and also Vayikra 18:5?"
+    refs = _extract_refs(text)
+    assert "Shemot 35:3" in refs
+    assert "Vayikra 18:5" in refs
+
+
+def test_extract_refs_handles_verse_ranges():
+    text = "Devarim 21:1-9 and Bereishit 1:1"
+    refs = _extract_refs(text)
+    assert any("Devarim 21:1" in r for r in refs)
+    assert "Bereishit 1:1" in refs
+
+
+def test_extract_refs_returns_empty_for_no_citations():
+    assert _extract_refs("This text has no Torah references.") == []