azazel_processor: commit 5 — orchestrator state machine (properties 5, 7, 10)

Pass 4 / Finding F, commit 5 of 10 from BUILD_AZAZEL.md. Largest
commit so far; pulls together vidui + shaliach + goral + stages and
adds the state machine + in-flight persistence.

THE STATE MACHINE (six checkpoints, persisted):

  STAGED       → AzazelStaged envelope tagged; payload held by shaliach
  CONFESSED    → vidui carried into the staged value
  ROUTED       → routing committed (always ERETZ_GEZERAH); authorized_by
                 + shaliach attribution recorded
  DESTROYED    → payload references cleared from shaliach;
                 gc.collect() called to release; payload becomes
                 unreachable from any orchestrator-visible code path
  ACKNOWLEDGED → out-of-band ack callback invoked (commit 9 wires
                 Chatzotzrot.signal(:moadim) here)
  RECORDED     → AzazelRecord persisted via callback (commit 7 wires
                 the cross-language L3 file-watcher bridge here)

After RECORDED completes, the in-flight checkpoint file is shredded —
the disposal is now permanent in L3, the in-flight artifact must not
linger.

PROPERTY COVERAGE:

  Property 5 (provisioned infrastructure / sukkot):
    Each checkpoint persists state to <in_flight_dir>/<case_ref>.json
    via write-then-rename (atomic against crash mid-write). list_in_flight
    returns partial disposals so resume tooling can pick them up.
    The Mishkan analog: the sukkot along the wilderness route where
    the ish itti can rest if the disposal halts.

  Property 7 (authorized routing):
    AzazelRecord.authorized_by carries the kohen_gadol identifier;
    AzazelRecord.shaliach carries the ish itti identifier. The
    routing destination is the ERETZ_GEZERAH sentinel (property 12).
    The routing decision is OWNED and traceable; the destination
    is not.

  Property 10 (hard termination):
    The DESTROYED checkpoint clears handler._payload_refs and calls
    gc.collect(). The AzazelRecord contains the vidui — what was
    disposed — but NOT the disposed payload itself. The to_storage_dict
    serialization explicitly excludes payload-shaped fields.

CREATED:

  src/azazel_processor/azazel_record.py    ~95 lines
    AzazelRecord                           — frozen dataclass; fields:
      case_ref, vidui, routing, authorized_by, shaliach,
      triggering_call, goral_outcome_id, disposed_at
      .to_storage_dict() — JSON-safe serialization for L3 + in-flight

  src/azazel_processor/azazel.py           ~225 lines
    AzazelInput                            — what is being disposed
    dispatch_for_disposal(input, *, handler, confess_with,
                          authorized_by, in_flight_dir,
                          acknowledger, recorder, goral_outcome_id)
                                           — the single legitimate
                                             entry point; walks the
                                             six checkpoints
    list_in_flight(in_flight_dir)          — surfaces partial disposals
    DEFAULT_IN_FLIGHT_DIR                  — /tmp/shorashim/azazel/in_flight

  test/test_azazel.py                      21 tests covering:
    HAPPY PATH:
    - dispatch returns AzazelRecord
    - record carries authorization + shaliach attribution
    - record routes to ERETZ_GEZERAH
    - record carries the full vidui

    PROPERTY 10 CI INVARIANTS:
    - record does not carry payload (by attribute name)
    - serialized record does not contain the payload's marker string
    - STRUCTURAL: AzazelRecord dataclass field set has no
      payload-shaped names (catches future regression)
    - STRUCTURAL: no recover/undispose/restore/rehydrate/retrieve
      function in azazel_processor public surface (catches
      future regression)
    - payload references cleared from shaliach at DESTROYED

    PROPERTY 5 CHECKPOINT TESTS:
    - in-flight file shredded after RECORDED
    - checkpoints persist incrementally (verified by inspecting the
      file at ack/rec callback fire-time)
    - list_in_flight returns partial disposals
    - list_in_flight empty when no disposals
    - persistence is atomic (file is valid JSON at every visit)

    ROLE GATE (property 4 still enforced):
    - dispatch refuses inactive handler
    - dispatch refuses non-IshItti

    FILENAME SAFETY:
    - case_ref with slash → ValueError
    - case_ref with NUL → ValueError

    AzazelRecord VALIDATION:
    - authorized_by required
    - shaliach required
    - to_storage_dict omits payload
    - frozen (cannot mutate)

DEFAULTS:

  acknowledger and recorder default to no-op callables. This is
  deliberate for commit 5 — the state machine works end-to-end without
  external side effects, and commits 7 (L3) and 9 (chatzotzrot) plug
  the real handlers into the same callback slots. Tests can supply
  their own to inspect mid-route state.

Tests: azazel_processor 109 passed (was 88, +21). korban_processor
unchanged at 203 passed, 1 skipped. No regressions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

azazel_processor/test/test_azazel.py

@@ -0,0 +1,429 @@
+"""
+Tests for the Azazel orchestrator (commit 5).
+
+Property coverage:
+  - Property 5  (provisioned infrastructure / sukkot): checkpoints
+                are persisted at every step; in-flight files exist
+                during the route and are shredded after RECORDED.
+  - Property 7  (authorized routing): AzazelRecord carries
+                authorized_by + shaliach + routing=ERETZ_GEZERAH.
+  - Property 10 (hard termination): AzazelRecord does NOT carry the
+                disposed payload; payload references in the shaliach
+                are cleared at the DESTROYED checkpoint.
+
+The structural tests in test_azazel_no_payload_field_in_record and
+test_azazel_no_recover_disposed_function are the load-bearing CI
+invariants — they catch future contributors who weaken property 10.
+"""
+
+import json
+import pathlib
+import tempfile
+
+import pytest
+
+from azazel_processor.azazel import (
+    AzazelInput,
+    dispatch_for_disposal,
+    list_in_flight,
+    _persist_checkpoint,
+)
+from azazel_processor.azazel_record import AzazelRecord
+from azazel_processor.eretz_gezerah import ERETZ_GEZERAH, is_eretz_gezerah
+from azazel_processor.shaliach import IshItti, ShaliachError
+from azazel_processor.stages import AzazelStage
+from azazel_processor.vidui import confess_empty_vital
+
+
+@pytest.fixture
+def tmp_in_flight():
+    """Per-test in-flight directory so tests don't share checkpoint state."""
+    with tempfile.TemporaryDirectory(prefix="azazel-test-") as d:
+        yield pathlib.Path(d)
+
+
+@pytest.fixture
+def active_handler():
+    h = IshItti("aharon-the-shaliach")
+    h.activate()
+    yield h
+    if h.is_active():
+        h.tevilah()
+
+
+def _make_input(case_ref="case-001", payload=None):
+    return AzazelInput(
+        case_ref=case_ref,
+        payload=payload if payload is not None else {"data": "would-be-disposed"},
+        classification="empty_vital",
+        triggering_call="external_offering.consult_external_model",
+    )
+
+
+def _vidui():
+    return confess_empty_vital(
+        "Claude response with no extractable Torah", confessor="kohen-gadol"
+    )
+
+
+# ============================================================
+# Happy path — full state machine
+# ============================================================
+
+
+def test_dispatch_returns_an_azazel_record(active_handler, tmp_in_flight):
+    record = dispatch_for_disposal(
+        _make_input(),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+    )
+    assert isinstance(record, AzazelRecord)
+
+
+def test_record_carries_authorization_and_shaliach_attribution(
+    active_handler, tmp_in_flight
+):
+    """Property 7: routing decision is OWNED."""
+    record = dispatch_for_disposal(
+        _make_input(),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol-aharon",
+        in_flight_dir=tmp_in_flight,
+    )
+    assert record.authorized_by == "kohen-gadol-aharon"
+    assert record.shaliach == "aharon-the-shaliach"
+
+
+def test_record_routes_to_eretz_gezerah(active_handler, tmp_in_flight):
+    """Property 7 + 12: destination is the canonical sentinel."""
+    record = dispatch_for_disposal(
+        _make_input(),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+    )
+    assert is_eretz_gezerah(record.routing)
+    assert record.routing is ERETZ_GEZERAH
+
+
+def test_record_carries_the_full_vidui(active_handler, tmp_in_flight):
+    v = _vidui()
+    record = dispatch_for_disposal(
+        _make_input(),
+        handler=active_handler,
+        confess_with=v,
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+    )
+    assert record.vidui is v
+    assert record.vidui.chatot == ("empty-vital offering: Claude response with no extractable Torah",)
+
+
+# ============================================================
+# Property 10: hard termination — no payload in the record
+# ============================================================
+
+
+def test_azazel_record_does_not_carry_payload(active_handler, tmp_in_flight):
+    """Property 10: the disposed thing is destroyed; only the vidui
+    remains. The AzazelRecord has no `payload` attribute and would-be
+    payloads are not findable via vars()."""
+    secret_payload = {"sensitive": "this-must-be-disposed-not-stored"}
+    record = dispatch_for_disposal(
+        _make_input(payload=secret_payload),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+    )
+
+    # The record must not have a payload field, by name
+    assert not hasattr(record, "payload")
+    # The record's serialized form must not contain the payload's
+    # marker string
+    serialized = json.dumps(record.to_storage_dict())
+    assert "this-must-be-disposed-not-stored" not in serialized
+
+
+def test_no_payload_field_in_azazel_record_dataclass():
+    """STRUCTURAL CI INVARIANT: AzazelRecord's field set must NOT
+    include `payload` (or any obvious alias). A future contributor
+    who adds a payload-bearing field violates property 10 and this
+    test fires before the regression ships."""
+    fields = set(AzazelRecord.__dataclass_fields__.keys())
+    forbidden = {"payload", "raw_payload", "disposed_payload", "data", "content"}
+    overlap = fields & forbidden
+    assert overlap == set(), (
+        f"AzazelRecord has payload-shaped field(s): {overlap}. Property "
+        f"10 (hard termination): the disposed payload MUST NOT be "
+        f"persisted on the record. Remove the field or reconsider."
+    )
+
+
+def test_no_recover_disposed_function_in_module():
+    """STRUCTURAL CI INVARIANT: there must be no recover_disposed,
+    undispose, restore_payload, or similar function in
+    azazel_processor's public surface. Property 10: no return path.
+    Eretz gezerah."""
+    import azazel_processor
+
+    public = [n for n in dir(azazel_processor) if not n.startswith("_")]
+    forbidden_substrings = ("recover", "undispose", "restore", "rehydrate", "retrieve")
+
+    offenders = [
+        name
+        for name in public
+        for sub in forbidden_substrings
+        if sub in name.lower()
+    ]
+
+    assert offenders == [], (
+        f"azazel_processor exposes recovery-shaped names: {offenders}. "
+        f"Property 10: there is no return path. Eretz gezerah."
+    )
+
+
+def test_payload_references_cleared_at_destroyed_checkpoint(
+    active_handler, tmp_in_flight
+):
+    """The shaliach's payload-refs list must be empty after dispatch
+    completes. Property 10 — the handler does not retain the disposed
+    payload."""
+    dispatch_for_disposal(
+        _make_input(payload={"big": "payload"}),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+    )
+    assert active_handler.held_payload_count() == 0
+
+
+# ============================================================
+# Property 5: provisioned infrastructure — sukkot persistence
+# ============================================================
+
+
+def test_in_flight_file_is_shredded_after_recorded_checkpoint(
+    active_handler, tmp_in_flight
+):
+    """After the disposal completes (RECORDED done), the in-flight
+    file is unlinked. The disposal is permanent in L3 (commit 7);
+    the in-flight artifact must not linger."""
+    dispatch_for_disposal(
+        _make_input(case_ref="case-shred-001"),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+    )
+    assert list_in_flight(tmp_in_flight) == []
+
+
+def test_checkpoint_is_persisted_at_each_stage(active_handler, tmp_in_flight):
+    """Verify state is persisted incrementally: capture the file at
+    each checkpoint via custom recorder/acknowledger callbacks."""
+    seen_at = {}
+
+    def at_acknowledged(record):
+        # By ACKNOWLEDGED, the in-flight file must contain
+        # last_completed = ACKNOWLEDGED's PRIOR step (DESTROYED).
+        # We've persisted ACKNOWLEDGED before invoking ack? No — we
+        # invoke ack AT this checkpoint, then persist. So when this
+        # callback fires, the file shows last_completed = DESTROYED.
+        # Actually, looking at the orchestrator: we invoke
+        # acknowledger() first, then persist ACKNOWLEDGED.
+        body = json.loads((tmp_in_flight / "case-cp-001.json").read_text())
+        seen_at["ack"] = body["last_completed"]
+
+    def at_recorded(record):
+        body = json.loads((tmp_in_flight / "case-cp-001.json").read_text())
+        seen_at["rec"] = body["last_completed"]
+
+    dispatch_for_disposal(
+        _make_input(case_ref="case-cp-001"),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+        acknowledger=at_acknowledged,
+        recorder=at_recorded,
+    )
+
+    # When acknowledger fired, last_completed should be DESTROYED
+    # (the ack callback runs BEFORE we persist the ACKNOWLEDGED
+    # checkpoint).
+    assert seen_at["ack"] == AzazelStage.DESTROYED.value
+    # When recorder fired, last_completed should be ACKNOWLEDGED.
+    assert seen_at["rec"] == AzazelStage.ACKNOWLEDGED.value
+
+
+def test_list_in_flight_returns_partial_disposals(tmp_in_flight):
+    """If we manually persist a partial state (simulating a crash),
+    list_in_flight surfaces it for resume."""
+    # Manually create an in-flight file at the CONFESSED checkpoint.
+    _persist_checkpoint(
+        tmp_in_flight,
+        "case-partial-001",
+        AzazelStage.CONFESSED,
+        {"shaliach": "aharon"},
+    )
+    inflight = list_in_flight(tmp_in_flight)
+    assert len(inflight) == 1
+    assert inflight[0]["case_ref"] == "case-partial-001"
+    assert inflight[0]["last_completed"] == "confessed"
+
+
+def test_list_in_flight_empty_when_no_disposals(tmp_in_flight):
+    assert list_in_flight(tmp_in_flight) == []
+
+
+def test_checkpoint_persistence_is_atomic(active_handler, tmp_in_flight):
+    """The persist path uses write-then-rename so a crash mid-write
+    leaves the previous checkpoint intact rather than a half-written
+    file. We can't easily simulate a crash, but we CAN verify the
+    final file is valid JSON at every checkpoint visit."""
+    files_seen = []
+
+    def at_recorded(record):
+        # Inspect the current file
+        if (tmp_in_flight / "case-atom-001.json").exists():
+            text = (tmp_in_flight / "case-atom-001.json").read_text()
+            files_seen.append(json.loads(text))  # raises if invalid
+
+    dispatch_for_disposal(
+        _make_input(case_ref="case-atom-001"),
+        handler=active_handler,
+        confess_with=_vidui(),
+        authorized_by="kohen-gadol",
+        in_flight_dir=tmp_in_flight,
+        recorder=at_recorded,
+    )
+    assert len(files_seen) == 1
+    assert files_seen[0]["last_completed"] == "acknowledged"
+
+
+# ============================================================
+# Role gate — property 4 still enforced
+# ============================================================
+
+
+def test_dispatch_refuses_inactive_handler(tmp_in_flight):
+    h = IshItti("aharon")
+    # Not activated
+    with pytest.raises(ShaliachError):
+        dispatch_for_disposal(
+            _make_input(),
+            handler=h,
+            confess_with=_vidui(),
+            authorized_by="kohen-gadol",
+            in_flight_dir=tmp_in_flight,
+        )
+
+
+def test_dispatch_refuses_non_ish_itti(tmp_in_flight):
+    with pytest.raises(ShaliachError):
+        dispatch_for_disposal(
+            _make_input(),
+            handler="not-a-handler",  # type: ignore[arg-type]
+            confess_with=_vidui(),
+            authorized_by="kohen-gadol",
+            in_flight_dir=tmp_in_flight,
+        )
+
+
+# ============================================================
+# Filename safety
+# ============================================================
+
+
+def test_dispatch_rejects_case_ref_with_slash(active_handler, tmp_in_flight):
+    bad = AzazelInput(
+        case_ref="../escape/attempt",
+        payload={},
+        classification="empty_vital",
+    )
+    with pytest.raises(ValueError):
+        dispatch_for_disposal(
+            bad,
+            handler=active_handler,
+            confess_with=_vidui(),
+            authorized_by="kohen-gadol",
+            in_flight_dir=tmp_in_flight,
+        )
+
+
+def test_dispatch_rejects_case_ref_with_nul(active_handler, tmp_in_flight):
+    bad = AzazelInput(
+        case_ref="case\0nul",
+        payload={},
+        classification="empty_vital",
+    )
+    with pytest.raises(ValueError):
+        dispatch_for_disposal(
+            bad,
+            handler=active_handler,
+            confess_with=_vidui(),
+            authorized_by="kohen-gadol",
+            in_flight_dir=tmp_in_flight,
+        )
+
+
+# ============================================================
+# AzazelRecord validation
+# ============================================================
+
+
+def test_record_requires_authorized_by():
+    with pytest.raises(ValueError):
+        AzazelRecord(
+            case_ref="x",
+            vidui=_vidui(),
+            routing=ERETZ_GEZERAH,
+            authorized_by="",
+            shaliach="aharon",
+        )
+
+
+def test_record_requires_shaliach():
+    with pytest.raises(ValueError):
+        AzazelRecord(
+            case_ref="x",
+            vidui=_vidui(),
+            routing=ERETZ_GEZERAH,
+            authorized_by="kohen-gadol",
+            shaliach="",
+        )
+
+
+def test_record_to_storage_dict_omits_payload():
+    """The serialization for L3 must not contain a payload field."""
+    record = AzazelRecord(
+        case_ref="case-storage-001",
+        vidui=_vidui(),
+        routing=ERETZ_GEZERAH,
+        authorized_by="kohen-gadol",
+        shaliach="aharon",
+    )
+    body = record.to_storage_dict()
+    assert "payload" not in body
+    assert "raw_payload" not in body
+    assert body["routing"] == "eretz_gezerah"
+    assert body["authorized_by"] == "kohen-gadol"
+    assert body["shaliach"] == "aharon"
+
+
+def test_record_is_frozen():
+    record = AzazelRecord(
+        case_ref="case-frozen-001",
+        vidui=_vidui(),
+        routing=ERETZ_GEZERAH,
+        authorized_by="kohen-gadol",
+        shaliach="aharon",
+    )
+    with pytest.raises((AttributeError, Exception)):
+        record.shaliach = "different"  # type: ignore[misc]