Scott KarpandClaude Opus 4.6
e223ab15b3
Vayikra 9:24 — vatetzei esh milifnei Hashem vatokhal al hamizbeach.
Phase 4 of the architecture is now complete. The remaining three steps
that were marked "blocked on external infrastructure" are built —
discipline-respecting, eish-zarah-aware, and tested.
## Step 22 — Korban Processor external model integration
(korban_processor/src/korban_processor/external_offering.py)
The single legitimate entry point for external model consultation.
Wraps Claude API calls through the offering protocol so external
output is treated as korban (raw material), never authoritative:
- semichah — provenance is tagged (avodah zarah Form 1 prevention)
- shechitah — assistant voice, hedging, formatting stripped
- zerikah — vital content extracted
- haktarah — converted to Prolog propositions
- route — sent to internal component per offering type
The raw Korban is preserved on the ProcessedKorban (gilui arayot
Form 3 prevention — input/output coupling never severed).
No internal component should call call_claude() directly.
11 tests, mocked external calls. Live API test opt-in via
SHORASHIM_LIVE_API=1.
## Step 24 — L2 Urim v'Tumim interface
(vav_pipeline/lib/vav_pipeline/urim_v_tumim.ex)
PRD §4.7: housed in the kohen gadol's breastplate; consulted only
on high-stakes decisions where normal deliberation is insufficient.
Bamidbar 27:21 — al-piv yetz'u v'al-piv yavo'u.
Three escalation steps, three honest outcomes:
1. resolved_by_specification — re-read produced an answer
2. resolved_by_external — external testimony recorded (NOT binding)
3. deferred_to_human — the system stops, asks for human judgment
Pre-condition gating refuses cases that didn't reach Sanhedrin teiku
first (no preempting deliberation) or that aren't flagged high-stakes
(ordinary teiku stays teiku). Records every consultation with steps
taken — there is no quiet escalation.
Per PRD: "The Urim v'Tumim never fabricates certainty where none
exists." When external also fails, the system stops. It does not
proceed on a guess.
Added to VavPipeline supervision tree. 14 tests.
## Step 21 — Kushya generator scaffolding
(kushya_generator/)
Directory layout per Architecture §3.6 with three layers:
- data/training_pairs/seed_pairs.jsonl — 10 hand-curated bootstrap
pairs covering 9 kushya types (scope_narrowing, internal_tension,
harmonization, exception_test, edge_case, scope_application,
rabbinic_constraint, method_question, subject_identification)
- data/prepare_training.py — validates the four kushya requirements
(specific Torah passage, identified weakness, question form,
non-trivial), produces train.jsonl + eval.jsonl with stratified
80/20 split by type
- fine_tune/config.yaml + train.py — QLoRA configuration and
training script. Operator's task — needs GPU. Script is
complete and self-contained.
- inference/kushya_api.py — three-backend interface
(fine_tuned → external → fallback). When fine-tuned model isn't
deployed, falls back to external via Korban Processor (Step 22).
The fallback is honest: returns no kushya rather than inventing
one. The Talmudic Engine's rule-based challenge.pl remains the
primary kushya source.
17 tests covering dataset validation, backend selection, and
reference extraction.
## .gitignore fix
The earlier "data/" line was too broad — it caught
kushya_generator/data/. Scoped to /data/ (root-level only).
## Test counts
- Elixir: 274 (was 260; +14 for UrimVTumim)
- Korban Python: 196 + 1 skipped (was 195; +1 module = 11 tests)
- Kushya Python: 17 (new)
- Haskell: 33 (unchanged)
- Prolog: 195 (unchanged)
- Total: 715 + 19 L3 excluded
## Phase 4 status
All 12 steps now have implementations:
- Steps 23, 25, 26, 27, 28, 29, 30, 31, 32: complete
- Steps 21, 22, 24: complete with discipline preserved
Step 21's fine-tuning run remains the operator's GPU task; the
pipeline is built such that a deployed adapter slots into the
inference backend without touching the Talmudic Engine or anything
above.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
125 lines
4.7 KiB
Python
125 lines
4.7 KiB
Python
"""
|
|
Tests for the kushya generator API.
|
|
|
|
These tests use mocked external callers; no real API requests are made.
|
|
"""
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
from unittest.mock import patch
|
|
|
|
# Add parent inference/ dir
|
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "inference"))
|
|
# Add the shorashim repo's korban_processor src to sys.path
|
|
sys.path.insert(
|
|
0,
|
|
str(Path(__file__).resolve().parent.parent.parent / "korban_processor" / "src"),
|
|
)
|
|
|
|
from kushya_api import KushyaResult, generate_kushya, _extract_refs # noqa: E402
|
|
|
|
|
|
# ============================================================
|
|
# Backend selection
|
|
# ============================================================
|
|
|
|
def test_fallback_returns_no_kushya_with_explanation():
|
|
"""The fallback path honestly reports no challenge — never invents one."""
|
|
result = generate_kushya("Some interpretation.", backend="fallback")
|
|
assert isinstance(result, KushyaResult)
|
|
assert result.kushya is None
|
|
assert result.backend == "fallback"
|
|
assert "does not invent" in (result.notes or "")
|
|
|
|
|
|
def test_fine_tuned_without_endpoint_falls_back_in_auto():
|
|
"""auto mode skips fine_tuned when no endpoint is configured, then
|
|
skips external when not available, ending in fallback."""
|
|
# We expect the Korban Processor's external model call to either succeed
|
|
# (if API key is present) or fail (if not) — either way, the result has
|
|
# a clear backend. Mock to ensure deterministic behavior.
|
|
from korban_processor.external_offering import consult_external_model # noqa: F401
|
|
|
|
with patch("kushya_api.consult_external_model") as mock_consult:
|
|
mock_consult.side_effect = RuntimeError("simulated API failure")
|
|
result = generate_kushya("Some interpretation.", backend="auto")
|
|
|
|
assert result.kushya is None
|
|
assert result.backend in ("fallback", "external")
|
|
|
|
|
|
def test_fine_tuned_explicit_without_endpoint_raises():
|
|
"""When backend='fine_tuned' is forced but no endpoint is configured,
|
|
we raise rather than silently falling through."""
|
|
import pytest
|
|
with pytest.raises(ValueError, match="fine_tuned_endpoint"):
|
|
generate_kushya("Some interpretation.", backend="fine_tuned")
|
|
|
|
|
|
# ============================================================
|
|
# External backend — mocked, exercises the full eish-zarah path
|
|
# ============================================================
|
|
|
|
def test_external_backend_returns_kushya_via_korban_processor():
|
|
"""When external consultation succeeds, the result is recorded with
|
|
'external' backend and a notes field that emphasizes it is testimony,
|
|
not ruling."""
|
|
from kushya_api import _try_external
|
|
|
|
fake_consultation = type("FakeConsultation", (), {})()
|
|
fake_consultation.raw_response = type(
|
|
"FakeResponse", (), {"content": "raw response with a question?"}
|
|
)()
|
|
fake_consultation.processed = type(
|
|
"FakeProcessed",
|
|
(),
|
|
{"vital_content": "What about Shemot 35:3 — does the prohibition extend to passive heating?"},
|
|
)()
|
|
|
|
with patch("kushya_api.consult_external_model", return_value=fake_consultation):
|
|
result = _try_external("Cooking on Shabbat is forbidden.", allow_fallback=True)
|
|
|
|
assert result.kushya
|
|
assert result.backend == "external"
|
|
assert "testimony" in (result.notes or "").lower()
|
|
assert "Shemot 35:3" in result.torah_refs
|
|
|
|
|
|
def test_external_backend_handles_empty_vital_content():
|
|
"""If zerikah extracts nothing vital from external response, we
|
|
record None — we do not pretend an empty kushya is real."""
|
|
from kushya_api import _try_external
|
|
|
|
fake_consultation = type("FakeConsultation", (), {})()
|
|
fake_consultation.raw_response = type("FakeResp", (), {"content": "Great question!"})()
|
|
fake_consultation.processed = type("FakeProc", (), {"vital_content": ""})()
|
|
|
|
with patch("kushya_api.consult_external_model", return_value=fake_consultation):
|
|
result = _try_external("Some interp.", allow_fallback=True)
|
|
|
|
assert result.kushya is None
|
|
assert result.backend == "external"
|
|
assert "no vital content" in (result.notes or "").lower()
|
|
|
|
|
|
# ============================================================
|
|
# Reference extraction
|
|
# ============================================================
|
|
|
|
def test_extract_refs_finds_torah_citations():
|
|
text = "What about Shemot 35:3 and also Vayikra 18:5?"
|
|
refs = _extract_refs(text)
|
|
assert "Shemot 35:3" in refs
|
|
assert "Vayikra 18:5" in refs
|
|
|
|
|
|
def test_extract_refs_handles_verse_ranges():
|
|
text = "Devarim 21:1-9 and Bereishit 1:1"
|
|
refs = _extract_refs(text)
|
|
assert any("Devarim 21:1" in r for r in refs)
|
|
assert "Bereishit 1:1" in refs
|
|
|
|
|
|
def test_extract_refs_returns_empty_for_no_citations():
|
|
assert _extract_refs("This text has no Torah references.") == []
|