vav_pipeline: commit 7 — L3 azazel_records + cross-language ingest GenServer
Pass 4 / Finding F, commit 7 of 10 from BUILD_AZAZEL.md. The
operator-internal half of property 9 (observable disposal). Crosses
the Python/Elixir boundary: the Python orchestrator drops completed
disposal records as JSON files; this commit's GenServer reads them
and persists to L3.
The public half (a /peah/azazel_log read endpoint) lands in commit 8;
the routing-from-Korban + Chatzotzrot wiring lands in commit 9. After
those, the Azazel Processor is end-to-end functional.
CREATED:
vav_pipeline/priv/repo/migrations/20260428000001_create_azazel_records.exs
azazel_records table — append-only, parallel to interpretations.
Columns: case_ref, vidui (jsonb), routing, authorized_by,
shaliach, triggering_call, goral_outcome_id, disposed_at.
Indexes on case_ref, disposed_at, shaliach.
Property-12-defending CHECK constraint:
routing_must_be_eretz_gezerah CHECK (routing = 'eretz_gezerah').
A future migration that wants to add a destination handle will
fail this check; the operator must explicitly justify removing
the constraint, which is the right friction.
Append-only enforcement matches interpretations: REVOKE
UPDATE/DELETE on the shorashim_app role.
vav_pipeline/lib/vav_pipeline/l3/azazel_record.ex
Ecto schema mirroring the Python AzazelRecord.to_storage_dict()
shape. Changeset validates:
- all required fields present
- routing == "eretz_gezerah" (mirrors the DB CHECK at app layer)
- vidui is a map with avonot/peshaim/chatot/confessor/
confessed_at slots present
- vidui's three categories not all empty (mirrors the Python
Vidui.__post_init__ EmptyConfessionError)
vav_pipeline/lib/vav_pipeline/azazel_ingest.ex ~190 lines
GenServer that polls a configurable pending-l3 directory
(default /tmp/shorashim/azazel/pending_l3). For each *.json file:
1. Read + JSON-parse
2. Translate Python serialization → Ecto attrs
3. Call recorder (default L3.Store.record_disposal/1)
4. On success, shred the source file (property-10 follow-through)
5. On failure, leave file in place for retry; log warning
Recorder is injectable for tests — mock recorder lets us verify
the pipeline without PostgreSQL.
Poll interval default 5s, configurable. Polling rather than
file_system events to keep the dependency surface minimal; the
Azazel cadence is exceptional content, not high-throughput.
Not added to the supervision tree in this commit — the producer
side (Python orchestrator dropping files) is wired in commit 9.
Adding to the supervisor before there's a producer means a
poller doing nothing useful. Operator can start manually for
early testing: VavPipeline.AzazelIngest.start_link(...).
CHANGED:
vav_pipeline/lib/vav_pipeline/l3/store.ex
+ record_disposal/1
Inserts an AzazelRecord. Returns {:ok, record} or
{:error, changeset}.
+ count_disposals/1
Total or since-a-timestamp count. Used by the public
observability endpoint (commit 8).
+ disposal_summary/1
Returns categorical info per disposal — case_ref, shaliach,
triggering_call, disposed_at, categories_filled (which of
avonot/peshaim/chatot had any items, NOT the items
themselves). PROPERTY-9 PRIVACY BOUNDARY at the public
side: items are operator-internal only, accessed via
future /kodesh/* endpoints.
TESTS:
vav_pipeline/test/azazel_ingest_test.exs 15 tests, no DB
Happy path:
- poll_now ingests valid file and shreds it
- multiple files processed in sorted order
- vidui preserved as map (jsonb shape)
Retry semantics:
- recorder error → file left in place + state.errors++
- malformed JSON → file left in place
- missing disposed_at → file left in place
State accounting:
- polls_completed, records_ingested, errors all tracked
- state survives multiple polls
- non-json files ignored
Changeset (no DB):
- valid attrs
- rejects routing != eretz_gezerah
- rejects vidui missing slots
- rejects vidui all-empty (untagged disposal)
- rejects missing required fields
- accepts optional triggering_call + goral_outcome_id
vav_pipeline/test/l3_store_test.exs +9 :l3-tagged
record_disposal:
- inserts valid disposal
- rejects routing != eretz_gezerah
- rejects all-empty vidui
- rejects missing required fields
Counts:
- count_disposals == 0 initially
- count_disposals after several disposals
- count_disposals(since) filters by timestamp
Summary:
- disposal_summary returns categorical info but NOT items
(privacy boundary verification)
- categories_filled accurately reflects vidui shape
Tests: vav_pipeline 353 passed, 0 failures, 33 excluded (was
329/24; +15 visible azazel_ingest + 9 :l3-tagged disposal store).
korban_processor + azazel_processor unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
6eb555a5cf
commit
2932fb7447
6 files changed
+997
-2
No files matched your search
@@ -0,0 +1,95 @@
|
||||
defmodule VavPipeline.Repo.Migrations.CreateAzazelRecords do
|
||||
@moduledoc """
|
||||
L3 — Azazel disposal records. Append-only, parallel to interpretations.
|
||||
|
||||
Vayikra 16: the disposed thing goes to eretz gezerah and does not
|
||||
return; the RECORD of the disposal remains as permanent oral
|
||||
tradition. The two-goat architecture: this table parallels
|
||||
interpretations the way the Azazel Processor parallels the Korban
|
||||
Processor.
|
||||
|
||||
PROPERTY 9 — observable disposal:
|
||||
The public fact of disposal is part of the implementation, not
|
||||
incidental to it. This table is the operator-internal half;
|
||||
the /peah/azazel_log endpoint (commit 8) is the public half.
|
||||
|
||||
PROPERTY 10 — hard termination:
|
||||
The DISPOSED PAYLOAD IS NOT IN THIS TABLE. Only the vidui (what
|
||||
was named) and the routing decision (always eretz_gezerah). A
|
||||
schema CHECK constraint enforces routing = 'eretz_gezerah' so
|
||||
a future migration cannot accidentally introduce a
|
||||
destination-handle column.
|
||||
|
||||
Append-only enforcement matches interpretations: REVOKE
|
||||
UPDATE/DELETE on the shorashim_app role.
|
||||
"""
|
||||
|
||||
use Ecto.Migration
|
||||
|
||||
def up do
|
||||
create table(:azazel_records) do
|
||||
# The opaque case_ref — content-addressable identifier of what
|
||||
# was disposed (NOT the content itself).
|
||||
add :case_ref, :text, null: false
|
||||
|
||||
# The vidui — operator-produced enumerative commit, three-fold
|
||||
# (avonot/peshaim/chatot) per Vayikra 16:21. Stored as JSONB
|
||||
# so the structure is queryable but the content is not parsed
|
||||
# by Postgres-side logic.
|
||||
add :vidui, :jsonb, null: false
|
||||
|
||||
# The routing — ALWAYS the literal string "eretz_gezerah". The
|
||||
# CHECK constraint below makes this structural.
|
||||
add :routing, :text, null: false
|
||||
|
||||
# Property 7 — authorized routing
|
||||
add :authorized_by, :text, null: false # kohen_gadol identifier
|
||||
add :shaliach, :text, null: false # ish itti identifier
|
||||
|
||||
# Audit metadata
|
||||
add :triggering_call, :text # which subsystem invoked
|
||||
add :goral_outcome_id, :text # if goral was cast, the
|
||||
# SHA-256 outcome digest
|
||||
# (audit-recomputable)
|
||||
|
||||
# When the disposal happened (per the orchestrator's clock, not
|
||||
# the L3 write clock — those can differ if the file watcher
|
||||
# picks up a backlog).
|
||||
add :disposed_at, :utc_datetime, null: false
|
||||
|
||||
timestamps(type: :utc_datetime, updated_at: false)
|
||||
end
|
||||
|
||||
# Property 12 / property 7: routing must be the canonical opaque
|
||||
# value. A future migration that wants to add a destination
|
||||
# handle will fail this check; the operator must explicitly
|
||||
# justify removing the constraint, which is the right friction.
|
||||
create constraint(
|
||||
:azazel_records,
|
||||
:routing_must_be_eretz_gezerah,
|
||||
check: "routing = 'eretz_gezerah'"
|
||||
)
|
||||
|
||||
create index(:azazel_records, [:case_ref])
|
||||
create index(:azazel_records, [:disposed_at])
|
||||
# Index on shaliach for operator queries ("what did this handler
|
||||
# dispose of?") — useful for tevilah audit and for property 4
|
||||
# accountability.
|
||||
create index(:azazel_records, [:shaliach])
|
||||
|
||||
# Append-only enforcement — same pattern as interpretations.
|
||||
execute """
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT FROM pg_roles WHERE rolname = 'shorashim_app') THEN
|
||||
REVOKE UPDATE, DELETE ON azazel_records FROM shorashim_app;
|
||||
END IF;
|
||||
END
|
||||
$$;
|
||||
"""
|
||||
end
|
||||
|
||||
def down do
|
||||
drop table(:azazel_records)
|
||||
end
|
||||
end
|
||||
Reference in new issue
Block a user