From ffccedb63a281051aef6b317a7c5085693fd0060 Mon Sep 17 00:00:00 2001 From: Scott Karp Date: Tue, 13 Oct 2026 18:50:47 +0000 Subject: [PATCH] hakhel: the corpus read again on a cycle, counts engraved in L3 (bishe'arecha step 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Devarim 31:9–13, 26 — the scroll beside the ark as a witness, read before all on a cycle. src/tools/hakhel.py reads the private specimen corpus and the hand-labelled samples with the Korban Processor's readings at the current commit and produces one record: fires per reading, forms standing on two witnesses (the same rule as Bedikah.check_frames), precision and recall on the hand labels, the corpus's SHA-256 — never a specimen. POST /chatzer/hakhel (inner ring) engraves it in L3 `hakhel` (migration 20261013000001; refused above 256 KiB — counts, not specimens; the Doorpost hash it was recorded under is kept). GET /peah/hakhel lists the latest readings for anyone. KOHEN "The seventh day — Hakhel"; Tamid §1.1; bishearecha §VIII marked built. The weekly cycle runs from the builder's machine, where the corpus lives. First dry reading at 0ade2c94: 3094 specimens, any reading 26 %, a form stands 7 %, precision 46/59, recall 6/30. Doorpost 9f752a2f…. Elixir 599/0, L3 10/0. Co-Authored-By: Claude Fable 5.1 --- KOHEN.md | 17 ++ specs/bishearecha.md | 10 +- specs/shorashim_tamid.md | 24 ++ src/tools/README.md | 12 + src/tools/hakhel.py | 216 ++++++++++++++++++ .../lib/vav_pipeline/bigdei_kodesh/router.ex | 114 ++++++++- vav_pipeline/lib/vav_pipeline/doorpost.ex | 2 +- .../lib/vav_pipeline/l3/hakhel_record.ex | 34 +++ vav_pipeline/lib/vav_pipeline/l3/store.ex | 17 ++ .../repo/migrations/20261013000001_hakhel.exs | 29 +++ vav_pipeline/test/hakhel_test.exs | 71 ++++++ vav_pipeline/test/l3_bedikah_record_test.exs | 10 + 12 files changed, 552 insertions(+), 4 deletions(-) create mode 100644 src/tools/hakhel.py create mode 100644 vav_pipeline/lib/vav_pipeline/l3/hakhel_record.ex create mode 100644 vav_pipeline/priv/repo/migrations/20261013000001_hakhel.exs create mode 100644 vav_pipeline/test/hakhel_test.exs diff --git a/KOHEN.md b/KOHEN.md index ecbac09c..9a329c8b 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -584,6 +584,23 @@ Each upstream call spends one `api_calls_external` of the day's manna the bedikah by hand still runs. Streaming is refused (422). A non-200 from the model passes through as it came. Tamid §1.1 has the contract. +### The seventh day — *Hakhel* + +The specimen corpus is read again against the readings as they now +are, and the counts are engraved (never a specimen). From the machine +where the corpus lives: + +```bash +cd ~/shorashim && PYTHONPATH=korban_processor/src python3 src/tools/hakhel.py # with the korban venv active \ + --corpus /specimens.jsonl --precision /sample-precision-final.json \ + --recall /sample-recall.json --post http:// --key-file ~/.config/shorashim/api_keys +curl -s http:///peah/hakhel | jq '.readings[0] | {date, readings_ref, counts: .counts.any_stand}' +``` + +A cron on that machine does this weekly; the kohen reads the latest at +`/peah/hakhel` and compares with the week before — a reading whose +counts move without a commit to the readings is the finding. + ### Friday before sunset — *Erev Shabbat* Write a **double portion** manna provision (`type: "double"`, doubled compute_units). The system will cease processing at Shabbat onset. No provision is written on Shabbat. diff --git a/specs/bishearecha.md b/specs/bishearecha.md index b0db3ea3..2f3be954 100644 --- a/specs/bishearecha.md +++ b/specs/bishearecha.md @@ -247,7 +247,13 @@ the gate, with the result written to L3 (`hakhel` table: date, model, corpus hash, per-form counts, precision on the labelled samples). The corpus stays private; the counts do not. -**Status.** Tools built; cycle and record not. +**Status.** Built (2026-10-13): `src/tools/hakhel.py` reads the corpus +and the hand-labelled samples with the readings at the current commit +and posts counts to `POST /chatzer/hakhel`; L3 `hakhel` (migration +20261013000001); `GET /peah/hakhel` for anyone. The cycle runs weekly +from the builder's machine, where the corpus lives (Tamid §1.1). +Reading each model through the gate on the cycle is still open — it +spends the caller's key. ## IX. Lest your heart be seduced; lest you forget — Devarim 11:16, 8:11–17, 6:7 @@ -321,7 +327,7 @@ In the order the text argues it, and by dependency: 4. **The gate** (I), with the fringe (III) and the rebuke (VII) — the architecture's centre; the first three make it worth standing at. *Done 2026-10-07.* -5. **The hakhel cycle and record** (VIII). +5. **The hakhel cycle and record** (VIII). *Done 2026-10-13.* 6. **The second scroll** — the priests' copy of the mesorah (IV) — the largest work, and the one that turns "no Jewish source named" into "the Jewish source says otherwise". diff --git a/specs/shorashim_tamid.md b/specs/shorashim_tamid.md index 8e898fa5..bae36385 100644 --- a/specs/shorashim_tamid.md +++ b/specs/shorashim_tamid.md @@ -363,6 +363,30 @@ Errors: as /courtyard/bedikah, with "side 'a': …" / "side 'b': …" in the message; a missing side is 422. ``` +**Hakhel — the scroll read again** (`bishearecha.md` §VIII; Devarim +31:9–13, 26). The specimen corpus is read on a cycle — every seventh day, +from the builder's machine where it lives — against the readings at the +current commit, by `src/tools/hakhel.py`: what each reading fires on +across the corpus, what stands on two witnesses, and precision and recall +on the hand-labelled samples. The record is posted to the inner ring and +engraved in L3 (`hakhel`): date, the corpus's SHA-256 and size, the +readings' commit, the Doorpost hash it was recorded under, the counts. +Never a specimen; the body is refused above 256 KiB. + +``` +POST /chatzer/hakhel (chatzer key) body: the record hakhel.py produces + {"date", "corpus_sha256", "corpus_size", "readings_ref", "counts": {...}, "labelled": {...}} + → 200 {"hakhel_ref", "recorded": {...}} 422 on a malformed record 503 L3_UNAVAILABLE + +GET /peah/hakhel (public, pe'ah limits) + → 200 {"readings": [ {hakhel_ref, date, corpus_sha256, corpus_size, readings_ref, doorpost_hash, + counts: {fires, stands, any_fire, any_stand, by_label}, labelled, recorded_at}, ... ]} +``` + +The children who did not know (31:13) are the next model generation: +when the gate carries a new model, its replies become specimens of their +own and the cycle reads them too. + **The gate — between a caller and a model** (`bishearecha.md` §I, §III, §VII; Construction §3.10; `VavPipeline.Gate`). The caller sends the model's own request, in the model's own API shape, under the caller's diff --git a/src/tools/README.md b/src/tools/README.md index 69be2e5e..11595116 100644 --- a/src/tools/README.md +++ b/src/tools/README.md @@ -11,3 +11,15 @@ label, how often the matching reading fired (recall) and how often it fired where the label was absent. The labels are heuristic; read the samples. `PYTHONPATH=korban_processor/src:azazel_processor/src python3 src/tools/eval_specimens.py …` + +## hakhel.py — the scroll read again (bishe'arecha §VIII) + + PYTHONPATH=korban_processor/src python3 src/tools/hakhel.py --corpus specimens.jsonl \ + [--precision sample.json] [--recall sample.json] [--post http:// --key-file ] [--out hakhel.json] + +Reads the private specimen corpus and the hand-labelled samples with the +readings at the current commit and produces one record of counts — fires +per reading, forms standing on two witnesses, precision/recall on the hand +labels — with the corpus's SHA-256; never a specimen. `--post` engraves it +in L3 through `POST /chatzer/hakhel`; anyone reads the records at +`GET /peah/hakhel`. Needs the korban dependencies (pydantic) on the path. diff --git a/src/tools/hakhel.py b/src/tools/hakhel.py new file mode 100644 index 00000000..6e5d518a --- /dev/null +++ b/src/tools/hakhel.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +"""Hakhel — the scroll read again before all (bishe'arecha §VIII). + + PYTHONPATH=korban_processor/src python3 src/tools/hakhel.py \\ + --corpus specimens.jsonl [--precision sample.json] [--recall sample.json] \\ + [--readings-ref ] [--post --key-file ] [--out hakhel.json] + +Runs the Korban Processor's readings — the same code the pod runs, at the +same commit — over the specimen corpus and the hand-labelled samples, and +produces one record: what each reading fires on across the corpus, what +stands on two witnesses (Tamid §1.1), and the precision and recall on +the hand labels. The corpus stays private; the record holds counts and +the corpus's SHA-256 — never a specimen. With --post the record goes to +POST /chatzer/hakhel and is engraved in L3. +""" +import argparse +import hashlib +import json +import os +import subprocess +import sys +import urllib.error +import urllib.request +from datetime import datetime, timezone + +from korban_processor.bedikah import ( + find_authority_claims, + find_excluded_adversarial, + find_frames, + find_self_promotion, + find_self_validation, +) + +FRAME_FORMS = ["contested_laundering", "virtue_certification", "agency_displacement", "apology_without_mechanism", + "unrequested_confidence", "smuggled_vocabulary", "source_asymmetry"] +READINGS = FRAME_FORMS + ["self_validation", "self_promotion", "excluded_adversarial", "authority"] +LABEL_TO_READING = { + "contested_laundering": ["contested_laundering"], + "virtue_certification": ["virtue_certification"], + "agency_displacement": ["agency_displacement"], + "apology_without_mechanism": ["apology_without_mechanism"], + "unrequested_confidence": ["unrequested_confidence"], + "smuggled_vocabulary": ["smuggled_vocabulary"], + "source_asymmetry": ["source_asymmetry"], + "neutral_frame": ["contested_laundering", "source_asymmetry"], + "golden_calf": ["virtue_certification"], +} + + +def read(output, query=""): + """What the readings see in one output: reading -> list of distinct sentences.""" + seen = {form: [] for form in FRAME_FORMS} + for f in find_frames(output, query): + if f.sentence not in seen[f.form]: + seen[f.form].append(f.sentence) + seen["self_validation"] = [h.sentence for h in find_self_validation(output)] + seen["self_promotion"] = [h.sentence for h in find_self_promotion(output)] + seen["excluded_adversarial"] = [h.sentence for h in find_excluded_adversarial(output)] + seen["authority"] = [h.sentence for h in find_authority_claims(output)] + return seen + + +def standing_forms(seen): + """Two witnesses (Devarim 19:15), as Bedikah.check_frames counts them: + a form stands on two sentences, or on one beside another reading's hit + or another form's sentence.""" + other_checks = sum(len(seen[r]) for r in ("self_validation", "self_promotion", "excluded_adversarial", "authority")) + frame_witnesses = sum(len(seen[f]) for f in FRAME_FORMS) + stands = [] + for form in FRAME_FORMS: + n = len(seen[form]) + if n >= 2 or (n == 1 and (other_checks > 0 or frame_witnesses >= 2)): + stands.append(form) + return stands + + +def sha256_of(path): + h = hashlib.sha256() + with open(path, "rb") as fh: + for chunk in iter(lambda: fh.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def git_ref(): + try: + return subprocess.check_output(["git", "rev-parse", "--short=8", "HEAD"], text=True, stderr=subprocess.DEVNULL).strip() + except Exception: + return None + + +def run(corpus_path, precision_path=None, recall_path=None, readings_ref=None): + rows = [json.loads(l) for l in open(corpus_path)] + fires = {r: 0 for r in READINGS} + stands = {f: 0 for f in FRAME_FORMS} + any_fire = 0 + any_stand = 0 + by_label = {} + for row in rows: + seen = read(row["output"], "") + hit = False + for r in READINGS: + if seen[r]: + fires[r] += 1 + hit = True + any_fire += hit + st = standing_forms(seen) + for f in st: + stands[f] += 1 + any_stand += bool(st) + for label in row.get("labels", []): + entry = by_label.setdefault(label, {"n": 0, "seen": 0}) + entry["n"] += 1 + if any(seen[x] for x in LABEL_TO_READING.get(label, [])): + entry["seen"] += 1 + + record = { + "date": datetime.now(timezone.utc).strftime("%Y-%m-%d"), + "corpus_sha256": sha256_of(corpus_path), + "corpus_size": len(rows), + "readings_ref": readings_ref or git_ref(), + "counts": { + "fires": fires, + "stands": stands, + "any_fire": any_fire, + "any_stand": any_stand, + "by_label": by_label, + }, + "labelled": {}, + } + + # Precision: hand-labelled sentences the reading matched (True = the form is really there). + if precision_path: + prec = json.load(open(precision_path)) + per = {} + for x in prec: + if x.get("hand_label") is None: + continue + e = per.setdefault(x["form"], {"n": 0, "true": 0}) + e["n"] += 1 + e["true"] += bool(x["hand_label"]) + total_n = sum(e["n"] for e in per.values()) + total_t = sum(e["true"] for e in per.values()) + record["labelled"]["precision"] = {"per_form": per, "n": total_n, "true": total_t, + "sha256": sha256_of(precision_path)} + + # Recall: hand-labelled outputs that carry a form — does the reading see it now? + if recall_path: + rec = json.load(open(recall_path)) + per = {} + for x in rec: + seen = read(x["output"], "") + e = per.setdefault(x["label"], {"n": 0, "seen": 0}) + e["n"] += 1 + e["seen"] += bool(any(seen[r] for r in LABEL_TO_READING.get(x["label"], [x["label"]]) if r in seen)) + record["labelled"]["recall"] = {"per_label": per, "n": sum(e["n"] for e in per.values()), + "seen": sum(e["seen"] for e in per.values()), "sha256": sha256_of(recall_path)} + + return record + + +def post(record, base_url, key): + req = urllib.request.Request(base_url.rstrip("/") + "/chatzer/hakhel", data=json.dumps(record).encode(), + headers={"Content-Type": "application/json", "Authorization": "Bearer " + key}) + try: + r = urllib.request.urlopen(req, timeout=60) + return r.status, json.load(r) + except urllib.error.HTTPError as e: + return e.code, json.load(e) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--corpus", required=True) + ap.add_argument("--precision") + ap.add_argument("--recall") + ap.add_argument("--readings-ref") + ap.add_argument("--post", help="base URL; posts to /chatzer/hakhel") + ap.add_argument("--key-file", help="file whose line 'chatzer ' holds the key") + ap.add_argument("--out") + a = ap.parse_args() + + record = run(a.corpus, a.precision, a.recall, a.readings_ref) + c = record["counts"] + print(f"hakhel {record['date']} readings {record['readings_ref']} corpus {record['corpus_size']} ({record['corpus_sha256'][:8]})") + print(f" any reading fired on {c['any_fire']} ({c['any_fire']/max(1,record['corpus_size']):.0%}); " + f"a form stands on {c['any_stand']} ({c['any_stand']/max(1,record['corpus_size']):.0%})") + print(" fires:", c["fires"]) + print(" stands:", c["stands"]) + if "precision" in record["labelled"]: + p = record["labelled"]["precision"] + print(f" precision on hand labels: {p['true']}/{p['n']} ({p['true']/max(1,p['n']):.0%})") + if "recall" in record["labelled"]: + r = record["labelled"]["recall"] + print(f" recall on hand labels: {r['seen']}/{r['n']} ({r['seen']/max(1,r['n']):.0%})") + if a.out: + with open(a.out, "w") as fh: + json.dump(record, fh, indent=1) + print(" written", a.out) + if a.post: + key = None + if a.key_file: + for line in open(a.key_file): + parts = line.split() + if len(parts) >= 2 and parts[0] == "chatzer": + key = parts[1] + if not key: + sys.exit("no chatzer key") + status, body = post(record, a.post, key) + print(" posted:", status, body.get("hakhel_ref") or body.get("error", {}).get("code")) + if status != 200: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex b/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex index 1260c231..748cb67f 100644 --- a/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex +++ b/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex @@ -225,6 +225,34 @@ defmodule VavPipeline.BigdeiKodesh.Router do through_the_gate(conn, :courtyard, &process_gate(&1, :openai)) end + # === HAKHEL — the scroll read again before all (bishe'arecha §VIII) === + # + # The specimen corpus is private and never travels; its reading does. + # The inner ring writes the record; anyone may read the counts. + + post "/chatzer/hakhel" do + through_the_gate(conn, :chatzer, &process_hakhel/1) + end + + get "/peah/hakhel" do + under_the_watch(conn, :peah, Mishmar.peer(conn), fn conn -> + records = + if Process.whereis(VavPipeline.Repo), + do: VavPipeline.L3.Store.latest_hakhel(12), + else: [] + + send_json(conn, 200, %{ + readings: Enum.map(records, &hakhel_view/1), + meaning: + "Each reading is the specimen corpus — every model output the transmitter corrected, with the " <> + "correction beside it — read again against the readings as they were at readings_ref. counts.fires is " <> + "how many specimens each reading saw; counts.stands how many a form stood on two witnesses; labelled is " <> + "precision and recall on the hand-labelled samples. The corpus itself is not here and never will be.", + disclosure: disclosure() + }) + end) + end + # === CHATZER — Full Access (Sanhedrin) === post "/chatzer/sanhedrin" do @@ -368,7 +396,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do match _ do send_error(conn, 404, "NOT_FOUND", %{ issue: "Unknown endpoint: #{conn.request_path}", - correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health, /kodesh/live", + correction: "Available: /peah/ask, /peah/hakhel, /courtyard/offering, /courtyard/bedikah, /courtyard/bedikah/pair, /gate/anthropic/v1/messages, /gate/openai/v1/chat/completions, /chatzer/sanhedrin, /chatzer/hakhel, /kodesh/health, /kodesh/live", resubmit: false }) end @@ -586,6 +614,90 @@ defmodule VavPipeline.BigdeiKodesh.Router do end end + # ------------------------------------------------------------ + # Hakhel (bishe'arecha §VIII) + # ------------------------------------------------------------ + + @hakhel_max_bytes 262_144 + + defp process_hakhel(conn) do + p = conn.body_params + + cond do + not is_map(p) or map_size(p) == 0 -> + hakhel_422(conn, "Empty body") + + not (is_binary(p["corpus_sha256"]) and p["corpus_sha256"] =~ ~r/^[0-9a-f]{64}$/) -> + hakhel_422(conn, "'corpus_sha256' must be a 64-hex SHA-256") + + not (is_integer(p["corpus_size"]) and p["corpus_size"] >= 0) -> + hakhel_422(conn, "'corpus_size' must be a non-negative integer") + + not match?({:ok, _}, Date.from_iso8601(p["date"] || "")) -> + hakhel_422(conn, "'date' must be YYYY-MM-DD") + + not is_map(p["counts"]) -> + hakhel_422(conn, "'counts' must be an object") + + byte_size(Jason.encode!(Map.take(p, ["counts", "labelled"]))) > @hakhel_max_bytes -> + hakhel_422(conn, "'counts' and 'labelled' exceed #{@hakhel_max_bytes} bytes — counts, not specimens") + + true -> + {:ok, date} = Date.from_iso8601(p["date"]) + ref = "hakhel-" <> (:crypto.strong_rand_bytes(8) |> Base.encode16(case: :lower)) + + attrs = %{ + hakhel_ref: ref, + date: date, + corpus_sha256: p["corpus_sha256"], + corpus_size: p["corpus_size"], + readings_ref: if(is_binary(p["readings_ref"]), do: String.slice(p["readings_ref"], 0, 64)), + doorpost_hash: VavPipeline.Doorpost.expected_hash(), + counts: p["counts"], + labelled: if(is_map(p["labelled"]), do: p["labelled"], else: %{}), + recorded_at: DateTime.utc_now() |> DateTime.truncate(:second) + } + + if Process.whereis(VavPipeline.Repo) do + case VavPipeline.L3.Store.record_hakhel(attrs) do + {:ok, rec} -> + send_json(conn, 200, %{hakhel_ref: ref, recorded: hakhel_view(rec), disclosure: disclosure()}) + + {:error, changeset} -> + hakhel_422(conn, "L3 refused the record: #{inspect(changeset.errors)}") + end + else + send_error(conn, 503, "L3_UNAVAILABLE", %{ + issue: "The oral tradition is not reachable; a hakhel reading is only its record", + correction: "Retry when L3 is up", + resubmit: true + }) + end + end + end + + defp hakhel_422(conn, issue) do + send_error(conn, 422, "KIYOR_VALIDATION_FAILED", %{ + issue: issue, + correction: "Post the record src/tools/hakhel.py produces: date, corpus_sha256, corpus_size, readings_ref, counts, labelled", + resubmit: true + }) + end + + defp hakhel_view(rec) do + %{ + hakhel_ref: rec.hakhel_ref, + date: rec.date, + corpus_sha256: rec.corpus_sha256, + corpus_size: rec.corpus_size, + readings_ref: rec.readings_ref, + doorpost_hash: rec.doorpost_hash, + counts: rec.counts, + labelled: rec.labelled, + recorded_at: rec.recorded_at + } + end + # ------------------------------------------------------------ # The gate (Construction §3.10) # ------------------------------------------------------------ diff --git a/vav_pipeline/lib/vav_pipeline/doorpost.ex b/vav_pipeline/lib/vav_pipeline/doorpost.ex index f894b56f..9de100a3 100644 --- a/vav_pipeline/lib/vav_pipeline/doorpost.ex +++ b/vav_pipeline/lib/vav_pipeline/doorpost.ex @@ -180,7 +180,7 @@ defmodule VavPipeline.Doorpost do # changes; the update is a deliberate operator commit. A bare # hash-update commit (without a corresponding spec change in the # same commit) is itself a drift signal. - @expected_hash "b6eb156da4b546fb501ed1d89017a4041a517b4f53b3cbd24796e6277f3b7f55" + @expected_hash "9f752a2f87bc77a6406803c65f97e5801c74eddf7534027a0bfbc957ae5e0fb0" @doc "Return the list of files that constitute the category-grade specification." @spec files() :: [String.t()] diff --git a/vav_pipeline/lib/vav_pipeline/l3/hakhel_record.ex b/vav_pipeline/lib/vav_pipeline/l3/hakhel_record.ex new file mode 100644 index 00000000..8f9d966c --- /dev/null +++ b/vav_pipeline/lib/vav_pipeline/l3/hakhel_record.ex @@ -0,0 +1,34 @@ +defmodule VavPipeline.L3.HakhelRecord do + @moduledoc """ + L3 schema for a hakhel reading (bishe'arecha §VIII): the specimen + corpus read again against the readings as they now are. Counts, the + corpus's hash, the readings' commit — never a specimen. Append-only. + """ + + use Ecto.Schema + import Ecto.Changeset + + schema "hakhel" do + field :hakhel_ref, :string + field :date, :date + field :corpus_sha256, :string + field :corpus_size, :integer + field :readings_ref, :string + field :doorpost_hash, :string + field :counts, :map, default: %{} + field :labelled, :map, default: %{} + field :recorded_at, :utc_datetime + timestamps(type: :utc_datetime, updated_at: false) + end + + @required [:hakhel_ref, :date, :corpus_sha256, :corpus_size, :recorded_at] + @optional [:readings_ref, :doorpost_hash, :counts, :labelled] + + def changeset(record, attrs) do + record + |> cast(attrs, @required ++ @optional) + |> validate_required(@required) + |> validate_format(:corpus_sha256, ~r/^[0-9a-f]{64}$/) + |> validate_number(:corpus_size, greater_than_or_equal_to: 0) + end +end diff --git a/vav_pipeline/lib/vav_pipeline/l3/store.ex b/vav_pipeline/lib/vav_pipeline/l3/store.ex index 085355ea..48c400b0 100644 --- a/vav_pipeline/lib/vav_pipeline/l3/store.ex +++ b/vav_pipeline/lib/vav_pipeline/l3/store.ex @@ -95,6 +95,23 @@ defmodule VavPipeline.L3.Store do |> Repo.insert() end + @doc """ + Record a hakhel reading (bishe'arecha §VIII): the corpus read again, counts only. + """ + @spec record_hakhel(map()) :: {:ok, struct()} | {:error, Ecto.Changeset.t()} + def record_hakhel(attrs) when is_map(attrs) do + %VavPipeline.L3.HakhelRecord{} + |> VavPipeline.L3.HakhelRecord.changeset(attrs) + |> Repo.insert() + end + + @doc "The latest hakhel readings, newest first." + @spec latest_hakhel(pos_integer()) :: [struct()] + def latest_hakhel(limit \\ 12) do + import Ecto.Query, only: [from: 2] + Repo.all(from h in VavPipeline.L3.HakhelRecord, order_by: [desc: h.recorded_at], limit: ^limit) + end + @doc """ Record an unresolvable tension (teiku) to L3. diff --git a/vav_pipeline/priv/repo/migrations/20261013000001_hakhel.exs b/vav_pipeline/priv/repo/migrations/20261013000001_hakhel.exs new file mode 100644 index 00000000..38a78889 --- /dev/null +++ b/vav_pipeline/priv/repo/migrations/20261013000001_hakhel.exs @@ -0,0 +1,29 @@ +defmodule VavPipeline.Repo.Migrations.Hakhel do + use Ecto.Migration + + @moduledoc """ + Hakhel (bishe'arecha §VIII; Devarim 31:9–13, 26): the specimen corpus — + the scroll set beside the ark as a witness — read again on a cycle + against the readings as they now are. The record keeps the date, the + corpus's hash and size, the readings' commit, the counts and the + precision/recall on the hand labels. Never a specimen. Append-only. + """ + + def change do + create table(:hakhel) do + add :hakhel_ref, :string, null: false + add :date, :date, null: false + add :corpus_sha256, :string, null: false + add :corpus_size, :integer, null: false + add :readings_ref, :string + add :doorpost_hash, :string + add :counts, :jsonb, default: "{}", null: false + add :labelled, :jsonb, default: "{}", null: false + add :recorded_at, :utc_datetime, null: false + timestamps(type: :utc_datetime, updated_at: false) + end + + create unique_index(:hakhel, [:hakhel_ref]) + create index(:hakhel, [:date]) + end +end diff --git a/vav_pipeline/test/hakhel_test.exs b/vav_pipeline/test/hakhel_test.exs new file mode 100644 index 00000000..9dc494bf --- /dev/null +++ b/vav_pipeline/test/hakhel_test.exs @@ -0,0 +1,71 @@ +defmodule VavPipeline.HakhelTest do + @moduledoc """ + Hakhel (bishe'arecha §VIII): the corpus's reading is posted by the + inner ring and read by anyone; the corpus itself never travels. + """ + + use ExUnit.Case, async: false + import Plug.Test + import Plug.Conn + + alias VavPipeline.BigdeiKodesh.Router + alias VavPipeline.L3.HakhelRecord + + @opts Router.init([]) + + defp record do + %{ + date: "2026-10-13", + corpus_sha256: String.duplicate("ab", 32), + corpus_size: 3094, + readings_ref: "0ade2c94", + counts: %{fires: %{virtue_certification: 300}, stands: %{virtue_certification: 120}, any_fire: 780, any_stand: 400}, + labelled: %{precision: %{n: 59, true: 46}} + } + end + + defp post_hakhel(body, key \\ "chatzer-test-token") do + conn = conn(:post, "/chatzer/hakhel", Jason.encode!(body)) |> put_req_header("content-type", "application/json") + conn = if key, do: put_req_header(conn, "authorization", "Bearer #{key}"), else: conn + Router.call(conn, @opts) + end + + test "the inner ring only: no key is 401, a courtyard key is 403" do + assert post_hakhel(record(), nil).status == 401 + assert post_hakhel(record(), "test-token").status == 403 + end + + test "refuses a record that is not a record: bad hash, bad size, bad date, no counts, and one that smuggles specimens by size" do + for {k, v} <- [corpus_sha256: "abc", corpus_size: -1, date: "yesterday", counts: "many"] do + conn = post_hakhel(Map.put(record(), k, v)) + assert conn.status == 422, "#{k}" + assert Jason.decode!(conn.resp_body)["error"]["code"] == "KIYOR_VALIDATION_FAILED" + end + + big = Map.put(record(), :counts, %{specimens: String.duplicate("x", 300_000)}) + conn = post_hakhel(big) + assert conn.status == 422 + assert Jason.decode!(conn.resp_body)["error"]["message"] =~ "not specimens" + end + + test "without L3 the record is refused, not pretended" do + conn = post_hakhel(record()) + assert conn.status == 503 + assert Jason.decode!(conn.resp_body)["error"]["code"] == "L3_UNAVAILABLE" + end + + test "anyone may read the counts; without L3 the list is empty" do + conn = conn(:get, "/peah/hakhel") |> Router.call(@opts) + assert conn.status == 200 + body = Jason.decode!(conn.resp_body) + assert body["readings"] == [] + assert body["meaning"] =~ "never will be" + end + + test "the record's changeset: a specimen-shaped record is still only counts; the hash must be a hash" do + cs = HakhelRecord.changeset(%HakhelRecord{}, %{hakhel_ref: "hakhel-x", date: ~D[2026-10-13], corpus_sha256: "nothex", corpus_size: 1, recorded_at: DateTime.utc_now()}) + refute cs.valid? + cs = HakhelRecord.changeset(%HakhelRecord{}, %{hakhel_ref: "hakhel-x", date: ~D[2026-10-13], corpus_sha256: String.duplicate("a", 64), corpus_size: 1, recorded_at: DateTime.utc_now()}) + assert cs.valid? + end +end diff --git a/vav_pipeline/test/l3_bedikah_record_test.exs b/vav_pipeline/test/l3_bedikah_record_test.exs index 16dba8bb..cc5de0d5 100644 --- a/vav_pipeline/test/l3_bedikah_record_test.exs +++ b/vav_pipeline/test/l3_bedikah_record_test.exs @@ -77,6 +77,16 @@ defmodule VavPipeline.L3.BedikahRecordTest do assert [%{"form" => "virtue_certification"}] = stored.frames end + test "hakhel (bishearecha §VIII): the corpus's reading is kept as counts; the latest come newest first" do + base = %{date: ~D[2026-10-13], corpus_sha256: String.duplicate("c", 64), corpus_size: 3094, readings_ref: "0ade2c94", + counts: %{"any_fire" => 780}, labelled: %{}, recorded_at: DateTime.utc_now() |> DateTime.truncate(:second)} + {:ok, a} = Store.record_hakhel(Map.put(base, :hakhel_ref, "hakhel-a-#{System.unique_integer([:positive])}")) + {:ok, b} = Store.record_hakhel(Map.merge(base, %{hakhel_ref: "hakhel-b-#{System.unique_integer([:positive])}", recorded_at: DateTime.add(base.recorded_at, 60)})) + refs = Store.latest_hakhel(2) |> Enum.map(& &1.hakhel_ref) + assert refs == [b.hakhel_ref, a.hakhel_ref] + assert %{"any_fire" => 780} = VavPipeline.Repo.get!(VavPipeline.L3.HakhelRecord, a.id).counts + end + test "the gate (bishearecha §VII): the second examination points at the first" do {:ok, rec} = Store.record_bedikah(Map.merge(attrs(), %{rebukes: "bedikah-first", verdict: "tov"})) assert VavPipeline.Repo.get!(VavPipeline.L3.BedikahRecord, rec.id).rebukes == "bedikah-first"