vav_pipeline: commit 7 — L3 azazel_records + cross-language ingest GenServer

Pass 4 / Finding F, commit 7 of 10 from BUILD_AZAZEL.md. The
operator-internal half of property 9 (observable disposal). Crosses
the Python/Elixir boundary: the Python orchestrator drops completed
disposal records as JSON files; this commit's GenServer reads them
and persists to L3.

The public half (a /peah/azazel_log read endpoint) lands in commit 8;
the routing-from-Korban + Chatzotzrot wiring lands in commit 9. After
those, the Azazel Processor is end-to-end functional.

CREATED:

  vav_pipeline/priv/repo/migrations/20260428000001_create_azazel_records.exs
    azazel_records table — append-only, parallel to interpretations.
    Columns: case_ref, vidui (jsonb), routing, authorized_by,
    shaliach, triggering_call, goral_outcome_id, disposed_at.
    Indexes on case_ref, disposed_at, shaliach.

    Property-12-defending CHECK constraint:
      routing_must_be_eretz_gezerah CHECK (routing = 'eretz_gezerah').
    A future migration that wants to add a destination handle will
    fail this check; the operator must explicitly justify removing
    the constraint, which is the right friction.

    Append-only enforcement matches interpretations: REVOKE
    UPDATE/DELETE on the shorashim_app role.

  vav_pipeline/lib/vav_pipeline/l3/azazel_record.ex
    Ecto schema mirroring the Python AzazelRecord.to_storage_dict()
    shape. Changeset validates:
      - all required fields present
      - routing == "eretz_gezerah" (mirrors the DB CHECK at app layer)
      - vidui is a map with avonot/peshaim/chatot/confessor/
        confessed_at slots present
      - vidui's three categories not all empty (mirrors the Python
        Vidui.__post_init__ EmptyConfessionError)

  vav_pipeline/lib/vav_pipeline/azazel_ingest.ex      ~190 lines
    GenServer that polls a configurable pending-l3 directory
    (default /tmp/shorashim/azazel/pending_l3). For each *.json file:
      1. Read + JSON-parse
      2. Translate Python serialization → Ecto attrs
      3. Call recorder (default L3.Store.record_disposal/1)
      4. On success, shred the source file (property-10 follow-through)
      5. On failure, leave file in place for retry; log warning

    Recorder is injectable for tests — mock recorder lets us verify
    the pipeline without PostgreSQL.

    Poll interval default 5s, configurable. Polling rather than
    file_system events to keep the dependency surface minimal; the
    Azazel cadence is exceptional content, not high-throughput.

    Not added to the supervision tree in this commit — the producer
    side (Python orchestrator dropping files) is wired in commit 9.
    Adding to the supervisor before there's a producer means a
    poller doing nothing useful. Operator can start manually for
    early testing: VavPipeline.AzazelIngest.start_link(...).

CHANGED:

  vav_pipeline/lib/vav_pipeline/l3/store.ex
    + record_disposal/1
        Inserts an AzazelRecord. Returns {:ok, record} or
        {:error, changeset}.

    + count_disposals/1
        Total or since-a-timestamp count. Used by the public
        observability endpoint (commit 8).

    + disposal_summary/1
        Returns categorical info per disposal — case_ref, shaliach,
        triggering_call, disposed_at, categories_filled (which of
        avonot/peshaim/chatot had any items, NOT the items
        themselves). PROPERTY-9 PRIVACY BOUNDARY at the public
        side: items are operator-internal only, accessed via
        future /kodesh/* endpoints.

TESTS:

  vav_pipeline/test/azazel_ingest_test.exs           15 tests, no DB
    Happy path:
      - poll_now ingests valid file and shreds it
      - multiple files processed in sorted order
      - vidui preserved as map (jsonb shape)
    Retry semantics:
      - recorder error → file left in place + state.errors++
      - malformed JSON → file left in place
      - missing disposed_at → file left in place
    State accounting:
      - polls_completed, records_ingested, errors all tracked
      - state survives multiple polls
      - non-json files ignored
    Changeset (no DB):
      - valid attrs
      - rejects routing != eretz_gezerah
      - rejects vidui missing slots
      - rejects vidui all-empty (untagged disposal)
      - rejects missing required fields
      - accepts optional triggering_call + goral_outcome_id

  vav_pipeline/test/l3_store_test.exs                +9 :l3-tagged
    record_disposal:
      - inserts valid disposal
      - rejects routing != eretz_gezerah
      - rejects all-empty vidui
      - rejects missing required fields
    Counts:
      - count_disposals == 0 initially
      - count_disposals after several disposals
      - count_disposals(since) filters by timestamp
    Summary:
      - disposal_summary returns categorical info but NOT items
        (privacy boundary verification)
      - categories_filled accurately reflects vidui shape

Tests: vav_pipeline 353 passed, 0 failures, 33 excluded (was
329/24; +15 visible azazel_ingest + 9 :l3-tagged disposal store).
korban_processor + azazel_processor unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Opus 4.7 committed 2026-04-28 09:16:21 -04:00
1 parent 6eb555a5cf
commit 2932fb7447
6 files changed
+997 -2

No files matched your search

@@ -0,0 +1,95 @@
defmodule VavPipeline.Repo.Migrations.CreateAzazelRecords do
@moduledoc """
L3 — Azazel disposal records. Append-only, parallel to interpretations.
Vayikra 16: the disposed thing goes to eretz gezerah and does not
return; the RECORD of the disposal remains as permanent oral
tradition. The two-goat architecture: this table parallels
interpretations the way the Azazel Processor parallels the Korban
Processor.
PROPERTY 9 — observable disposal:
The public fact of disposal is part of the implementation, not
incidental to it. This table is the operator-internal half;
the /peah/azazel_log endpoint (commit 8) is the public half.
PROPERTY 10 — hard termination:
The DISPOSED PAYLOAD IS NOT IN THIS TABLE. Only the vidui (what
was named) and the routing decision (always eretz_gezerah). A
schema CHECK constraint enforces routing = 'eretz_gezerah' so
a future migration cannot accidentally introduce a
destination-handle column.
Append-only enforcement matches interpretations: REVOKE
UPDATE/DELETE on the shorashim_app role.
"""
use Ecto.Migration
def up do
create table(:azazel_records) do
# The opaque case_ref — content-addressable identifier of what
# was disposed (NOT the content itself).
add :case_ref, :text, null: false
# The vidui — operator-produced enumerative commit, three-fold
# (avonot/peshaim/chatot) per Vayikra 16:21. Stored as JSONB
# so the structure is queryable but the content is not parsed
# by Postgres-side logic.
add :vidui, :jsonb, null: false
# The routing — ALWAYS the literal string "eretz_gezerah". The
# CHECK constraint below makes this structural.
add :routing, :text, null: false
# Property 7 — authorized routing
add :authorized_by, :text, null: false # kohen_gadol identifier
add :shaliach, :text, null: false # ish itti identifier
# Audit metadata
add :triggering_call, :text # which subsystem invoked
add :goral_outcome_id, :text # if goral was cast, the
# SHA-256 outcome digest
# (audit-recomputable)
# When the disposal happened (per the orchestrator's clock, not
# the L3 write clock — those can differ if the file watcher
# picks up a backlog).
add :disposed_at, :utc_datetime, null: false
timestamps(type: :utc_datetime, updated_at: false)
end
# Property 12 / property 7: routing must be the canonical opaque
# value. A future migration that wants to add a destination
# handle will fail this check; the operator must explicitly
# justify removing the constraint, which is the right friction.
create constraint(
:azazel_records,
:routing_must_be_eretz_gezerah,
check: "routing = 'eretz_gezerah'"
)
create index(:azazel_records, [:case_ref])
create index(:azazel_records, [:disposed_at])
# Index on shaliach for operator queries ("what did this handler
# dispose of?") — useful for tevilah audit and for property 4
# accountability.
create index(:azazel_records, [:shaliach])
# Append-only enforcement — same pattern as interpretations.
execute """
DO $$
BEGIN
IF EXISTS (SELECT FROM pg_roles WHERE rolname = 'shorashim_app') THEN
REVOKE UPDATE, DELETE ON azazel_records FROM shorashim_app;
END IF;
END
$$;
"""
end
def down do
drop table(:azazel_records)
end
end