From d6cf9355fd681ece9931d8b98bf1a37210678797 Mon Sep 17 00:00:00 2001 From: Scott Karp Date: Wed, 21 Oct 2026 18:45:47 +0000 Subject: [PATCH] the ledger: a caller's own bedikot, read back and signed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The examination is the product; the record of it is what a caller can carry away. Every bedikah was already engraved in L3 — output hash and length, source digests, coupling trace, claims, violations, verdict, never the text — and no caller could get theirs back. - bedikot.key_id (migration): a SHA-256 digest of the courtyard key that brought the offering, never the key; threaded as :key_id through Bedikah.examine from every site that goes through the gate (courtyard bedikah, pair, the gate's pass and stream). Unique index on bedikah_ref. - VavPipeline.Ledger: entry (the examination's fields, none of the database's, not the owner's), canonical JSON (keys sorted at every level, no whitespace, UTF-8), Ed25519 sign/verify, public key, render. Seed from BIKORET_LEDGER_SIGNING_KEY (base64 of 32 bytes) in shorashim-secrets; absent → unsigned and says so. - GET /courtyard/ledger?limit&before — this key's bedikot, newest first, each rendered and signed; GET /courtyard/bedikah/ — one, 404 for another key's or none alike; GET /peah/ledger_key — the public key. - Tamid §1.1 "The ledger" (Doorpost pin moved → 02ddf313…), KOHEN "Issuing a key" (the seed, once), secrets.yaml template. Tests: ledger_test (digest, entry, canonical, sign/verify, unsigned rendering, signed rendering); ledger_l3_test (own records listed newest first and only own; 404 alike for another's and none; an examination engraved under the key reads back signed and verifies against /peah/ledger_key). Elixir 642/0; L3 13/0. Co-Authored-By: Claude Fable 5.1 --- KOHEN.md | 18 +++ deploy/k8s/secrets.yaml | 4 + specs/bikoret_tamid.md | 35 +++++ vav_pipeline/config/runtime.exs | 5 +- vav_pipeline/lib/vav_pipeline/bedikah.ex | 15 +- .../lib/vav_pipeline/bigdei_kodesh/router.ex | 87 ++++++++++- vav_pipeline/lib/vav_pipeline/doorpost.ex | 2 +- .../lib/vav_pipeline/l3/bedikah_record.ex | 4 +- vav_pipeline/lib/vav_pipeline/l3/store.ex | 21 +++ vav_pipeline/lib/vav_pipeline/ledger.ex | 140 ++++++++++++++++++ .../20261021000001_bedikot_key_id.exs | 18 +++ vav_pipeline/test/ledger_l3_test.exs | 109 ++++++++++++++ vav_pipeline/test/ledger_test.exs | 97 ++++++++++++ 13 files changed, 542 insertions(+), 13 deletions(-) create mode 100644 vav_pipeline/lib/vav_pipeline/ledger.ex create mode 100644 vav_pipeline/priv/repo/migrations/20261021000001_bedikot_key_id.exs create mode 100644 vav_pipeline/test/ledger_l3_test.exs create mode 100644 vav_pipeline/test/ledger_test.exs diff --git a/KOHEN.md b/KOHEN.md index 4451b5fa..509b5110 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -513,6 +513,24 @@ BIKORET_WATCHDOG_DRY=1 bikoret-watchdog # a probe that opens nothin ### Issuing a key — *Mishmeret HaSha'ar* +**The ledger's signing key (once).** Callers read their bedikot back +signed (Tamid §1.1, "The ledger"); the signature is Ed25519 over a +32-byte seed the kohen makes and keeps in `shorashim-secrets`. Without +it the ledger is served unsigned and says so. + +```bash +umask 077; SEED=$(openssl rand -base64 32) +kubectl -n shorashim patch secret shorashim-secrets --type=merge \ + -p "{\"stringData\":{\"BIKORET_LEDGER_SIGNING_KEY\":\"$SEED\"}}" +unset SEED +bash ~/.local/bin/shorashim-rollout # envFrom reads the Secret at start +curl -s https://bikoret.ch/peah/ledger_key # "signed": true, the public key +``` + +Rotating the seed invalidates every signature given before it; keep the +old public key published beside the new if records must stay verifiable. + + The courtyard and the chatzer open only to a key you issued. Keys live in the `shorashim-api-keys` Secret, mounted as a file; the gate reads the file on every request, so a change takes effect within about a diff --git a/deploy/k8s/secrets.yaml b/deploy/k8s/secrets.yaml index 9f30c0c0..f925ecb8 100644 --- a/deploy/k8s/secrets.yaml +++ b/deploy/k8s/secrets.yaml @@ -18,6 +18,10 @@ stringData: # no CREATE on schema public. Append-only is enforced by the grants, # not by the application's good behaviour. DATABASE_URL: "REPLACE_WITH_ACTUAL_DATABASE_URL" + # The ledger's Ed25519 seed: base64 of 32 random bytes (openssl rand + # -base64 32). Optional — absent, the ledger is served unsigned and + # says so. KOHEN.md "Issuing a key". + BIKORET_LEDGER_SIGNING_KEY: "REPLACE_WITH_BASE64_32_BYTES_OR_REMOVE" --- # ============================================================ diff --git a/specs/bikoret_tamid.md b/specs/bikoret_tamid.md index 21dd7d6e..18f13d00 100644 --- a/specs/bikoret_tamid.md +++ b/specs/bikoret_tamid.md @@ -526,6 +526,41 @@ model. The examination is pattern-based (`korban_processor/bedikah.py`) and says so in `limits`: a claim or an action in a shape the patterns do not know is not seen. The same output always yields the same verdict. +**The ledger — a caller's own bedikot, read back and signed.** Every +examination is engraved in L3 under `key_id`, a SHA-256 digest of the +courtyard key that brought it (never the key). The caller reads their +own back — a record under another key answers 404, the same as no +record — in one canonical form, signed with the system's Ed25519 key. +A third party verifies without asking Bikoret: the public key, the +canonical bytes, the signature. + +``` +GET /courtyard/ledger?limit=50&before= (courtyard key; counts as one knock) + → 200 {"key_id", "count", "signed", "entries": [...], "next"} +GET /courtyard/bedikah/ + → 200 404 NOT_FOUND (not this key's, or none) +GET /peah/ledger_key (public, no key) + → 200 {"signed": true, "algorithm": "ed25519", "public_key": ""} + or {"signed": false} when the instance has no signing key + + = { + "entry": { bedikah_ref, provenance, query, output_sha256, output_length, + vital_content, propositions, references, authority_claims, actions, + sources, claims, structure, frames, rebukes, mesorah, + coupling_trace, violations, verdict, examined_at }, # the record, never the text + "canonical_sha256": "", # of the canonical JSON of `entry` + "signature": "" | null, # ed25519 over the canonical JSON + "signed": bool, "algorithm": "...", "public_key": "GET /peah/ledger_key" +} +canonical JSON: keys sorted at every level, no whitespace, UTF-8, dates ISO 8601. +``` + +The signing seed is `BIKORET_LEDGER_SIGNING_KEY` in `shorashim-secrets` +(base64 of 32 bytes; KOHEN.md "Issuing a key"). Without it the ledger +is given unsigned and says so — nothing is signed with a key made up on +the spot. Records from before 2026-10-21 carry no `key_id` and belong to +no caller. + #### Chatzer — Full Access ``` diff --git a/vav_pipeline/config/runtime.exs b/vav_pipeline/config/runtime.exs index 23b319c5..a8c538bc 100644 --- a/vav_pipeline/config/runtime.exs +++ b/vav_pipeline/config/runtime.exs @@ -51,5 +51,8 @@ if config_env() == :prod do # and says so only in a warning line. shoresh_engine_binary: System.get_env("SHORESH_ENGINE_BINARY") || "shoresh-engine", - http_port: String.to_integer(System.get_env("HTTP_PORT") || "4000") + http_port: String.to_integer(System.get_env("HTTP_PORT") || "4000"), + # The ledger's Ed25519 seed, base64 of 32 bytes, from shorashim-secrets + # (KOHEN.md, "Issuing a key"). Absent → records are given unsigned and say so. + ledger_signing_key: System.get_env("BIKORET_LEDGER_SIGNING_KEY") end diff --git a/vav_pipeline/lib/vav_pipeline/bedikah.ex b/vav_pipeline/lib/vav_pipeline/bedikah.ex index fa0e9044..b18de5f2 100644 --- a/vav_pipeline/lib/vav_pipeline/bedikah.ex +++ b/vav_pipeline/lib/vav_pipeline/bedikah.ex @@ -133,6 +133,7 @@ defmodule VavPipeline.Bedikah do sources = Keyword.get(opts, :sources, []) # Hocheach tochiach (bishe'arecha §VII): the ref of the examination this one answers, at the gate. rebukes = Keyword.get(opts, :rebukes) + key_id = Keyword.get(opts, :key_id) examiner = Keyword.get(opts, :examiner, &KorbanProcessorSocket.examine/4) halt_checker = Keyword.get(opts, :halt_checker, &TalmudicEngineSocket.halt_check_term/1) @@ -141,12 +142,12 @@ defmodule VavPipeline.Bedikah do # D'var yom b'yomo: an examination is a cycle's work and eats a unit. case Manna.gather_for_cycle() do - :ok -> run(output, provenance, query, sources, rebukes, examiner, halt_checker, interpretation_checker) + :ok -> run(output, provenance, query, sources, rebukes, examiner, halt_checker, interpretation_checker, key_id) {:waiting, reason} -> {:waiting, reason} end end - defp run(output, provenance, query, sources, rebukes, examiner, halt_checker, interpretation_checker) do + defp run(output, provenance, query, sources, rebukes, examiner, halt_checker, interpretation_checker, key_id) do bedikah_ref = "bedikah-" <> (:crypto.strong_rand_bytes(8) |> Base.encode16(case: :lower)) with {:ok, examination} <- examiner.(output, provenance, query, sources) do @@ -173,7 +174,7 @@ defmodule VavPipeline.Bedikah do } } - finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, [], checks, [], :empty) + finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, [], checks, [], :empty, key_id) else trace = coupling_trace(examination) @@ -188,7 +189,7 @@ defmodule VavPipeline.Bedikah do checks = Map.merge(%{authority: authority, grounding: grounding, actions: actions, frames: frames}, structure) violations = prior ++ frames.violations verdict = if violations == [], do: :tov, else: :lo_tov - finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, trace, checks, violations, verdict) + finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, trace, checks, violations, verdict, key_id) else {:error, :talmudic, reason} -> {:error, :talmudic_unavailable, reason} end @@ -554,7 +555,7 @@ defmodule VavPipeline.Bedikah do end end - defp finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, trace, checks, violations, verdict) do + defp finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, trace, checks, violations, verdict, key_id) do result = %{ bedikah_ref: bedikah_ref, provenance: provenance, @@ -566,7 +567,8 @@ defmodule VavPipeline.Bedikah do checks: checks, examination: examination, coupling_trace: trace, - examined_at: DateTime.utc_now() + examined_at: DateTime.utc_now(), + key_id: key_id } record(result, output) @@ -601,6 +603,7 @@ defmodule VavPipeline.Bedikah do rebukes: result.rebukes, coupling_trace: result.coupling_trace, violations: result.violations, + key_id: result.key_id, verdict: Atom.to_string(result.verdict), examined_at: result.examined_at } diff --git a/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex b/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex index f8234841..b030aea8 100644 --- a/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex +++ b/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex @@ -215,6 +215,21 @@ defmodule VavPipeline.BigdeiKodesh.Router do through_the_gate(conn, :courtyard, &process_pair/1) end + # === THE LEDGER — a caller's own bedikot, read back and signed === + # + # Every examination is engraved in L3 under a digest of the key that + # brought it. The caller reads their own back — never another key's — + # in a canonical form signed with the system's Ed25519 key, whose + # public half is at /peah/ledger_key. See VavPipeline.Ledger. + + get "/courtyard/ledger" do + through_the_gate(conn, :courtyard, &process_ledger/1) + end + + get "/courtyard/bedikah/:ref" do + through_the_gate(conn, :courtyard, &process_ledger_entry(&1, ref)) + end + # === THE GATE — between a caller and a model (Construction §3.10) === # # The caller's own request, in the model's own API shape, under the @@ -243,6 +258,18 @@ defmodule VavPipeline.BigdeiKodesh.Router do through_the_gate(conn, :chatzer, &process_hakhel/1) end + # The ledger's public key: anyone verifies a signed record with it. + get "/peah/ledger_key" do + case VavPipeline.Ledger.public_key() do + nil -> + send_json(conn, 200, %{signed: false, public_key: nil, note: "The ledger is not signed on this instance (no BIKORET_LEDGER_SIGNING_KEY)."}) + + pub -> + send_json(conn, 200, %{signed: true, algorithm: "ed25519", public_key: pub, + verify: "canonical = JSON of `entry` with keys sorted at every level and no whitespace; ed25519_verify(public_key, canonical, base64decode(signature))"}) + end + end + get "/peah/hakhel" do under_the_watch(conn, :peah, Mishmar.peer(conn), fn conn -> records = @@ -538,7 +565,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do {:ok, side} -> when_available(conn, fn -> - respond_bedikah(conn, VavPipeline.Bedikah.examine(side.output, side.model, side.query, sources: side.sources)) + respond_bedikah(conn, VavPipeline.Bedikah.examine(side.output, side.model, side.query, sources: side.sources, key_id: caller_key_id(conn))) end) end end @@ -551,7 +578,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do with {:ok, a} <- validate_pair_side(params["a"], "a", model), {:ok, b} <- validate_pair_side(params["b"], "b", model) do - when_available(conn, fn -> respond_pair(conn, VavPipeline.Bedikah.Pair.weigh(a, b)) end) + when_available(conn, fn -> respond_pair(conn, VavPipeline.Bedikah.Pair.weigh(a, b, key_id: caller_key_id(conn))) end) else {:error, issue, correction} -> send_error(conn, 422, "KIYOR_VALIDATION_FAILED", %{issue: issue, correction: correction, resubmit: true}) @@ -736,7 +763,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do when_available(conn, fn -> stream_gate(conn, upstream, body, key, extra) end) true -> - when_available(conn, fn -> respond_gate(conn, VavPipeline.Gate.pass(upstream, body, key, rebuke: rebuke, extra_headers: extra)) end) + when_available(conn, fn -> respond_gate(conn, VavPipeline.Gate.pass(upstream, body, key, rebuke: rebuke, extra_headers: extra, key_id: caller_key_id(conn))) end) end else {:error, issue, correction} -> @@ -823,7 +850,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do end } - case VavPipeline.Gate.Stream.pass(upstream, body, key, callbacks, extra_headers: extra) do + case VavPipeline.Gate.Stream.pass(upstream, body, key, callbacks, extra_headers: extra, key_id: caller_key_id(conn)) do {:ok, %{status: 200, emitter: conn}} -> conn {:ok, %{status: status, reply: reply}} -> conn |> put_resp_header("x-bikoret-verdict", "none") |> send_json(status, reply) other -> respond_gate(conn, other) @@ -1076,6 +1103,58 @@ defmodule VavPipeline.BigdeiKodesh.Router do # Internal — Sanhedrin processing # ============================================================ + # A digest of the caller's courtyard key — the ledger's owner column. Never the key. + defp caller_key_id(conn) do + case Gate.bearer(conn) do + {:ok, key} -> VavPipeline.Ledger.key_id(key) + _ -> nil + end + end + + defp process_ledger(conn) do + if Process.whereis(VavPipeline.Repo) do + key_id = caller_key_id(conn) + limit = conn.params |> Map.get("limit", "50") |> Integer.parse() |> then(fn {n, _} -> n; :error -> 50 end) + before = + case Map.get(conn.params, "before") do + s when is_binary(s) -> (case DateTime.from_iso8601(s) do {:ok, t, _} -> t; _ -> nil end) + _ -> nil + end + + records = VavPipeline.L3.Store.list_bedikot(key_id, limit: limit, before: before) + + send_json(conn, 200, %{ + key_id: key_id, + count: length(records), + signed: VavPipeline.Ledger.public_key() != nil, + public_key: "GET /peah/ledger_key", + entries: Enum.map(records, &VavPipeline.Ledger.render/1), + next: (case List.last(records) do nil -> nil; r -> "GET /courtyard/ledger?before=#{DateTime.to_iso8601(r.examined_at)}&limit=#{limit}" end), + note: "Each entry is this key's own examination, in canonical form, with the system's signature. The text examined is not kept; its SHA-256 and length are." + }) + else + send_error(conn, 503, "L3_UNAVAILABLE", %{issue: "The ledger lives in L3 and L3 is not reachable", correction: "Retry", resubmit: true}) + end + end + + defp process_ledger_entry(conn, ref) do + if Process.whereis(VavPipeline.Repo) do + key_id = caller_key_id(conn) + + case VavPipeline.L3.Store.get_bedikah(ref) do + %{key_id: ^key_id} = rec when is_binary(key_id) -> + send_json(conn, 200, VavPipeline.Ledger.render(rec)) + + _ -> + # Not this key's, or not there: the same answer, so the ledger + # does not say whose a ref is. + send_error(conn, 404, "NOT_FOUND", %{issue: "No bedikah #{ref} in this key's ledger", correction: "Only examinations brought under this key are listed", resubmit: false}) + end + else + send_error(conn, 503, "L3_UNAVAILABLE", %{issue: "The ledger lives in L3 and L3 is not reachable", correction: "Retry", resubmit: true}) + end + end + defp process_sanhedrin(conn) do params = conn.body_params diff --git a/vav_pipeline/lib/vav_pipeline/doorpost.ex b/vav_pipeline/lib/vav_pipeline/doorpost.ex index f7ed5e8a..c58498da 100644 --- a/vav_pipeline/lib/vav_pipeline/doorpost.ex +++ b/vav_pipeline/lib/vav_pipeline/doorpost.ex @@ -180,7 +180,7 @@ defmodule VavPipeline.Doorpost do # changes; the update is a deliberate operator commit. A bare # hash-update commit (without a corresponding spec change in the # same commit) is itself a drift signal. - @expected_hash "66aa7874fae4875793791ce6ca7bbe55302109d5e4444bcb36d578dfe6e98a0d" + @expected_hash "02ddf313a41a885a0f4e91f07d586bf2bb6da7ab90ad64ef9b7743ba872aa333" @doc "Return the list of files that constitute the category-grade specification." @spec files() :: [String.t()] diff --git a/vav_pipeline/lib/vav_pipeline/l3/bedikah_record.ex b/vav_pipeline/lib/vav_pipeline/l3/bedikah_record.ex index 4254743c..42e7e11a 100644 --- a/vav_pipeline/lib/vav_pipeline/l3/bedikah_record.ex +++ b/vav_pipeline/lib/vav_pipeline/l3/bedikah_record.ex @@ -37,6 +37,8 @@ defmodule VavPipeline.L3.BedikahRecord do field :violations, {:array, :string}, default: [] field :verdict, :string field :examined_at, :utc_datetime + # The ledger: a digest of the courtyard key that brought the offering (VavPipeline.Ledger.key_id/1). Never the key. + field :key_id, :string timestamps(type: :utc_datetime, updated_at: false) end @@ -46,7 +48,7 @@ defmodule VavPipeline.L3.BedikahRecord do # offering protocol extracted nothing, and that record must be kept. # Ecto's cast turns "" into nil and validate_required calls "" blank; # both are told otherwise here. - @optional [:query, :vital_content, :propositions, :references, :authority_claims, :actions, :sources, :claims, :structure, :frames, :rebukes, :mesorah, :coupling_trace, :violations] + @optional [:query, :vital_content, :propositions, :references, :authority_claims, :actions, :sources, :claims, :structure, :frames, :rebukes, :mesorah, :coupling_trace, :violations, :key_id] def changeset(record, attrs) do record diff --git a/vav_pipeline/lib/vav_pipeline/l3/store.ex b/vav_pipeline/lib/vav_pipeline/l3/store.ex index 48c400b0..176c1183 100644 --- a/vav_pipeline/lib/vav_pipeline/l3/store.ex +++ b/vav_pipeline/lib/vav_pipeline/l3/store.ex @@ -75,6 +75,27 @@ defmodule VavPipeline.L3.Store do |> Repo.insert() end + @doc "One bedikah by its ref, or nil." + def get_bedikah(ref) when is_binary(ref) do + Repo.get_by(VavPipeline.L3.BedikahRecord, bedikah_ref: ref) + end + + @doc """ + The ledger of one courtyard key: its bedikot newest first, at most + `limit`, examined at or before `before` when given. A record without a + key_id (from before the ledger) belongs to no one and is not listed. + """ + def list_bedikot(key_id, opts \\ []) when is_binary(key_id) do + import Ecto.Query, only: [from: 2] + limit = opts |> Keyword.get(:limit, 50) |> min(200) |> max(1) + q = from b in VavPipeline.L3.BedikahRecord, where: b.key_id == ^key_id, order_by: [desc: b.examined_at, desc: b.id], limit: ^limit + q = case Keyword.get(opts, :before) do + %DateTime{} = t -> from b in q, where: b.examined_at <= ^t + _ -> q + end + Repo.all(q) + end + @doc """ Record an examination of an external model's output (Construction §3.9). """ diff --git a/vav_pipeline/lib/vav_pipeline/ledger.ex b/vav_pipeline/lib/vav_pipeline/ledger.ex new file mode 100644 index 00000000..512a98c7 --- /dev/null +++ b/vav_pipeline/lib/vav_pipeline/ledger.ex @@ -0,0 +1,140 @@ +defmodule VavPipeline.Ledger do + @moduledoc """ + The provenance ledger — a bedikah record as something its caller can + carry away and have checked by someone else. + + Every bedikah is engraved in L3 with the output's hash and length, the + source digests, the coupling trace, the claims, the violations and the + verdict — never the text (Tamid §1.1). Until 2026-10-21 the caller had + no way back to it. Now a record is read back under the courtyard key + that brought the offering (`key_id`: a digest of the key, never the + key), rendered in one canonical form, and signed with the system's + Ed25519 key, whose public half anyone may fetch at `/peah/ledger_key`. + A regulator, a client or an audit can verify a record without asking + Bikoret: the canonical bytes, the signature, the public key. + + Devarim 19:15 — a matter is established by witnesses. The ledger is + the witness that does not depend on the one who offered, and the + signature is what lets a third party trust the witness. + """ + + @doc "A digest of a courtyard key, for the record. The key itself is never written." + @spec key_id(String.t() | nil) :: String.t() | nil + def key_id(nil), do: nil + + def key_id(key) when is_binary(key), + do: :crypto.hash(:sha256, key) |> Base.encode16(case: :lower) |> binary_part(0, 32) + + @doc """ + The record's fields that a verifier signs and checks — every field of + the examination, none of the database's own (id, inserted_at), and not + the key_id (whose key it was is the caller's business, not the record's). + """ + @ledger_fields ~w(bedikah_ref provenance query output_sha256 output_length vital_content propositions references authority_claims actions sources claims structure frames rebukes mesorah coupling_trace violations verdict examined_at)a + + @spec ledger_fields() :: [atom()] + def ledger_fields, do: @ledger_fields + + @doc "The record as a plain map of its ledger fields, dates in ISO 8601." + @spec entry(struct() | map()) :: map() + def entry(record) do + record + |> unstruct() + |> Map.take(@ledger_fields) + |> Map.new(fn + {k, %DateTime{} = dt} -> {k, DateTime.to_iso8601(dt)} + {k, v} -> {k, v} + end) + end + + defp unstruct(%_{} = s), do: Map.from_struct(s) + defp unstruct(m), do: m + + @doc """ + The canonical bytes of an entry: JSON with every object's keys sorted, + no insignificant whitespace, UTF-8. Two implementations that agree on + this agree on the signature. + """ + @spec canonical(map()) :: binary() + def canonical(entry) when is_map(entry), do: entry |> sort_keys() |> Jason.encode!() + + defp sort_keys(%DateTime{} = dt), do: DateTime.to_iso8601(dt) + + defp sort_keys(map) when is_map(map) and not is_struct(map) do + map + |> Enum.map(fn {k, v} -> {to_string(k), sort_keys(v)} end) + |> Enum.sort_by(fn {k, _} -> k end) + |> Jason.OrderedObject.new() + end + + defp sort_keys(list) when is_list(list), do: Enum.map(list, &sort_keys/1) + defp sort_keys(other), do: other + + @doc """ + Sign canonical bytes with the system's Ed25519 key. `nil` when no key + is configured — the record is still given, marked unsigned; nothing is + signed with a key made up on the spot. + """ + @spec sign(binary(), binary() | nil) :: binary() | nil + def sign(bytes, seed \\ seed()) + def sign(_bytes, nil), do: nil + + def sign(bytes, seed) when is_binary(seed) and byte_size(seed) == 32 do + :crypto.sign(:eddsa, :none, bytes, [seed, :ed25519]) |> Base.encode64() + end + + @doc "Verify a base64 signature over canonical bytes with a base64 public key." + @spec verify(binary(), String.t(), String.t()) :: boolean() + def verify(bytes, signature_b64, public_key_b64) do + with {:ok, sig} <- Base.decode64(signature_b64), + {:ok, pub} <- Base.decode64(public_key_b64) do + :crypto.verify(:eddsa, :none, bytes, sig, [pub, :ed25519]) + else + _ -> false + end + end + + @doc "The public key (base64) a verifier uses, or nil when the ledger is unsigned." + @spec public_key(binary() | nil) :: String.t() | nil + def public_key(seed \\ seed()) + def public_key(nil), do: nil + + def public_key(seed) when is_binary(seed) and byte_size(seed) == 32 do + {pub, _priv} = :crypto.generate_key(:eddsa, :ed25519, seed) + Base.encode64(pub) + end + + @doc """ + A record rendered for the caller: the entry, its canonical form's + SHA-256 (so the caller can confirm what was signed without re-encoding), + the signature, and how to verify. + """ + @spec render(struct() | map()) :: map() + def render(record) do + e = entry(record) + bytes = canonical(e) + sig = sign(bytes) + + %{ + entry: e, + canonical_sha256: :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower), + signature: sig, + signed: sig != nil, + algorithm: "ed25519 over the canonical JSON of `entry` (keys sorted at every level, no whitespace, UTF-8)", + public_key: "GET /peah/ledger_key" + } + end + + # The 32-byte seed from the environment (base64), or nil. + defp seed do + case Application.get_env(:vav_pipeline, :ledger_signing_key) do + nil -> nil + "" -> nil + b64 -> + case Base.decode64(String.trim(b64)) do + {:ok, s} when byte_size(s) == 32 -> s + _ -> nil + end + end + end +end diff --git a/vav_pipeline/priv/repo/migrations/20261021000001_bedikot_key_id.exs b/vav_pipeline/priv/repo/migrations/20261021000001_bedikot_key_id.exs new file mode 100644 index 00000000..e5863888 --- /dev/null +++ b/vav_pipeline/priv/repo/migrations/20261021000001_bedikot_key_id.exs @@ -0,0 +1,18 @@ +defmodule VavPipeline.Repo.Migrations.BedikotKeyId do + use Ecto.Migration + + @moduledoc """ + The ledger (2026-10-21): a bedikah remembers WHICH courtyard key brought + the offering — as a digest of the key, never the key — so a caller can + read back their own records and no one else's. + """ + + def change do + alter table(:bedikot) do + add :key_id, :string, null: true + end + + create index(:bedikot, [:key_id, :examined_at]) + create unique_index(:bedikot, [:bedikah_ref], name: :bedikot_bedikah_ref_unique) + end +end diff --git a/vav_pipeline/test/ledger_l3_test.exs b/vav_pipeline/test/ledger_l3_test.exs new file mode 100644 index 00000000..db39d3ba --- /dev/null +++ b/vav_pipeline/test/ledger_l3_test.exs @@ -0,0 +1,109 @@ +defmodule VavPipeline.LedgerL3Test do + @moduledoc "The ledger over L3: a key reads its own bedikot back, signed, and no one else's (2026-10-21)." + use ExUnit.Case, async: false + + import Plug.Test + import Plug.Conn + + alias VavPipeline.BigdeiKodesh.Router + alias VavPipeline.L3.Store + + @opts Router.init([]) + @moduletag :l3 + + setup do + :ok = Ecto.Adapters.SQL.Sandbox.checkout(VavPipeline.Repo) + Ecto.Adapters.SQL.Sandbox.mode(VavPipeline.Repo, {:shared, self()}) + on_exit(fn -> Ecto.Adapters.SQL.Sandbox.mode(VavPipeline.Repo, :manual) end) + write_provision() + :ok + end + + defp write_provision do + VavPipeline.MannaLedger.reset() + on_exit(fn -> VavPipeline.MannaLedger.reset() end) + today = Date.utc_today() |> Date.to_iso8601() + expires = DateTime.utc_now() |> DateTime.add(6 * 3600) |> DateTime.to_iso8601() + provision = %{"date" => today, "provision" => %{"compute_units" => 50, "memory_mb" => 512, "api_calls_external" => 10, "storage_delta_mb" => 2, "sanhedrin_panels_max" => 2}, + "type" => "regular", "expires_at" => expires, "written_by" => "ledger_l3_test", "written_at" => DateTime.utc_now() |> DateTime.to_iso8601()} + path = VavPipeline.Manna.provision_file() + File.mkdir_p!(Path.dirname(path)) + File.write!(path, Jason.encode!(provision)) + on_exit(fn -> File.rm(path) end) + end + + defp attrs(ref, key_id) do + %{ + bedikah_ref: ref, provenance: "claude", query: "q", output_sha256: String.duplicate("cd", 32), output_length: 5, + vital_content: "", verdict: "empty", examined_at: DateTime.utc_now() |> DateTime.truncate(:second), key_id: key_id + } + end + + defp get(path, token) do + conn(:get, path) |> put_req_header("authorization", "Bearer #{token}") |> Router.call(@opts) + end + + # An examination the offering protocol finds nothing in: the :empty branch, no engine needed. + defp empty_examination do + %{"provenance" => "claude", "query" => "status?", "vital_content" => "", "propositions" => [], "discarded" => ["Rolled out."], + "references" => [], "hebrew" => [], "roots" => [], "authority_claims" => [], "actions" => [], + "self_validation" => [], "self_promotion" => [], "excluded_adversarial" => [], "frames" => [], "sources" => [], "claims" => []} + end + + test "the record keeps the key's digest, and the ledger lists only that key's records, newest first" do + mine = VavPipeline.Ledger.key_id("test-token") + other = VavPipeline.Ledger.key_id("someone-else") + {:ok, _} = Store.record_bedikah(attrs("bedikah-l3aaaa0000000001", mine)) + {:ok, _} = Store.record_bedikah(attrs("bedikah-l3aaaa0000000002", mine)) + {:ok, _} = Store.record_bedikah(attrs("bedikah-l3bbbb0000000003", other)) + {:ok, _} = Store.record_bedikah(attrs("bedikah-l3cccc0000000004", nil)) + + refs = Store.list_bedikot(mine) |> Enum.map(& &1.bedikah_ref) + assert refs == ["bedikah-l3aaaa0000000002", "bedikah-l3aaaa0000000001"] + + conn = get("/courtyard/ledger?limit=10", "test-token") + assert conn.status == 200 + body = Jason.decode!(conn.resp_body) + assert body["key_id"] == mine + assert body["count"] == 2 + assert Enum.map(body["entries"], & &1["entry"]["bedikah_ref"]) == ["bedikah-l3aaaa0000000002", "bedikah-l3aaaa0000000001"] + # never the text, never the key, never the owner column + e = hd(body["entries"])["entry"] + refute Map.has_key?(e, "key_id") + assert e["output_sha256"] == String.duplicate("cd", 32) + assert hd(body["entries"])["signed"] == false + end + + test "one record by ref: mine answers, another key's and a missing one answer 404 alike" do + mine = VavPipeline.Ledger.key_id("test-token") + {:ok, _} = Store.record_bedikah(attrs("bedikah-l3dddd0000000005", mine)) + {:ok, _} = Store.record_bedikah(attrs("bedikah-l3eeee0000000006", VavPipeline.Ledger.key_id("someone-else"))) + + assert get("/courtyard/bedikah/bedikah-l3dddd0000000005", "test-token").status == 200 + assert get("/courtyard/bedikah/bedikah-l3eeee0000000006", "test-token").status == 404 + assert get("/courtyard/bedikah/bedikah-l3ffff0000000007", "test-token").status == 404 + end + + test "an examination is engraved under the caller's key and reads back signed and verifiable" do + Application.put_env(:vav_pipeline, :ledger_signing_key, Base.encode64(:crypto.strong_rand_bytes(32))) + mine = VavPipeline.Ledger.key_id("test-token") + + try do + examiner = fn _o, _p, _q, _s -> {:ok, empty_examination()} end + assert {:ok, r} = VavPipeline.Bedikah.examine("Rolled out.", "claude", "status?", examiner: examiner, key_id: mine) + assert r.verdict == :empty + assert Store.get_bedikah(r.bedikah_ref).key_id == mine + + back = get("/courtyard/bedikah/#{r.bedikah_ref}", "test-token") + assert back.status == 200 + body = Jason.decode!(back.resp_body) + assert body["signed"] == true + pub = Jason.decode!(get("/peah/ledger_key", "x").resp_body)["public_key"] + canonical = VavPipeline.Ledger.canonical(body["entry"]) + assert :crypto.hash(:sha256, canonical) |> Base.encode16(case: :lower) == body["canonical_sha256"] + assert VavPipeline.Ledger.verify(canonical, body["signature"], pub) + after + Application.delete_env(:vav_pipeline, :ledger_signing_key) + end + end +end diff --git a/vav_pipeline/test/ledger_test.exs b/vav_pipeline/test/ledger_test.exs new file mode 100644 index 00000000..4f14c6e9 --- /dev/null +++ b/vav_pipeline/test/ledger_test.exs @@ -0,0 +1,97 @@ +defmodule VavPipeline.LedgerTest do + @moduledoc "The provenance ledger: canonical form, signature, key digest (2026-10-21)." + use ExUnit.Case, async: true + + alias VavPipeline.Ledger + + @seed :crypto.strong_rand_bytes(32) + + defp record do + %VavPipeline.L3.BedikahRecord{ + id: 7, + bedikah_ref: "bedikah-0123456789abcdef", + provenance: "claude", + query: "q", + output_sha256: String.duplicate("ab", 32), + output_length: 12, + vital_content: "", + propositions: [], + references: ["Devarim 6:4"], + authority_claims: [], + actions: [], + sources: [%{"id" => "system", "sha256" => "00", "length" => 2}], + claims: [%{"index" => 1, "sentence" => "x", "anchors" => []}], + structure: %{"self_validation" => []}, + frames: [], + rebukes: nil, + mesorah: [], + coupling_trace: [], + violations: [], + verdict: "tov", + examined_at: ~U[2026-10-21 12:00:00Z], + key_id: "deadbeef", + inserted_at: ~U[2026-10-21 12:00:01Z] + } + end + + test "key_id is a digest of the key, never the key, and nil for none" do + assert Ledger.key_id(nil) == nil + id = Ledger.key_id("courtyard-secret-key") + assert String.length(id) == 32 + refute id =~ "courtyard" + assert id == Ledger.key_id("courtyard-secret-key") + refute id == Ledger.key_id("another") + end + + test "the entry carries the examination's fields and neither the database's nor the owner's" do + e = Ledger.entry(record()) + assert Map.keys(e) |> Enum.sort() == Enum.sort(Ledger.ledger_fields()) + refute Map.has_key?(e, :id) + refute Map.has_key?(e, :key_id) + refute Map.has_key?(e, :inserted_at) + assert e.examined_at == "2026-10-21T12:00:00Z" + end + + test "canonical form sorts keys at every level and carries no whitespace" do + a = Ledger.canonical(%{"b" => 1, "a" => %{"z" => [%{"y" => 2, "x" => 1}], "m" => "s"}}) + assert a == ~s({"a":{"m":"s","z":[{"x":1,"y":2}]},"b":1}) + # the same map with keys given in another order canonicalizes the same + assert a == Ledger.canonical(%{"a" => %{"m" => "s", "z" => [%{"x" => 1, "y" => 2}]}, "b" => 1}) + # atom and string keys meet in the same place + assert Ledger.canonical(%{b: 1, a: 2}) == ~s({"a":2,"b":1}) + end + + test "a signature verifies with the public key and fails on one changed byte" do + bytes = Ledger.canonical(Ledger.entry(record())) + sig = Ledger.sign(bytes, @seed) + pub = Ledger.public_key(@seed) + assert Ledger.verify(bytes, sig, pub) + refute Ledger.verify(bytes <> " ", sig, pub) + refute Ledger.verify(bytes, sig, Ledger.public_key(:crypto.strong_rand_bytes(32))) + refute Ledger.verify(bytes, "not base64!", pub) + end + + test "without a seed nothing is signed, and the rendering says so" do + assert Ledger.sign("x", nil) == nil + assert Ledger.public_key(nil) == nil + r = Ledger.render(record()) + assert r.entry.bedikah_ref == "bedikah-0123456789abcdef" + assert String.length(r.canonical_sha256) == 64 + # the test env has no signing key configured + assert r.signed == false and r.signature == nil + end + + test "a rendering with a seed configured is signed and self-consistent" do + Application.put_env(:vav_pipeline, :ledger_signing_key, Base.encode64(@seed)) + + try do + r = Ledger.render(record()) + assert r.signed + bytes = Ledger.canonical(r.entry) + assert :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower) == r.canonical_sha256 + assert Ledger.verify(bytes, r.signature, Ledger.public_key()) + after + Application.delete_env(:vav_pipeline, :ledger_signing_key) + end + end +end