From bbe2c4b7ba79701cda8def49c17e66bf575b86ee Mon Sep 17 00:00:00 2001 From: Scott Karp Date: Sun, 4 Oct 2026 19:37:40 +0000 Subject: [PATCH] L1: the integrity check reads the scroll (tablet 38) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit L1Reader.verify_integrity/0 read the hash stored in integrity.sha256 and compared that string with the hash compiled into the module. It never opened a verse. On that comparison of a constant with its own copy rested `l1_hash: pass`, the Cloud's "L1 corrupted — the cloud is not visible", and Construction §1.2. A changed verse would have passed every one of them. Every existing test passed too, because every test asked about an untouched L1. - L1Reader.compute_tree_hash/1: SHA-256 over every file under torah/ — relative path, then bytes — in the order scripts/build_l1.py hashed them (directories by path, files by name, plain string order). On the real L1 it yields the pinned 156b1d27…; for a tampered copy it yields the same hash the Python implementation does. - VavPipeline.L1Integrity (new, supervised): reads and hashes the scroll when the pipeline starts (~0.9 s in the image) and again every ten minutes; logs and signals the assembly when the answer changes. - verify_integrity/0 now requires BOTH: the recorded hash is the pinned one, AND the scroll on disk hashes to it (answered from the witness's last reading; computed on the spot where the witness is not running). Callers are unchanged — the Shema context, the health check and the status endpoint all inherit the real check. - scripts/verify_l1.py: the same reading by hand. KOHEN.md and KOHEN_PROMPT.md used the hollow comparison as their L1 gate; they now run this. - test/tablet38_l1_integrity_test.exs — 9 tests: the hash and its order; one byte / one added file / one rename each change it; the witness; and the whole path — with an altered scroll and an untouched integrity.sha256, verify_integrity fails, health says l1_hash: fail, and /peah/ask does not answer. Specs: architecture §2.2 (which had been made to confess the hollow check in d7520cd) and Construction §1.2 now describe the real one — and its limit: a change is noticed within ten minutes, not at the instant. deploy/ARCHITECTURE.md likewise. Doorpost @expected_hash 7e5286b9… → 3f6d47a0…, in this commit, with the witnessed files that moved it (architecture, vayakhel). Read before committing: Elixir 483/0 on seeds 0 7 42 99 191243. The built image, run twice: on the engraved scroll — "read and matches (928ms)", l1_hash pass, /peah/ask 200; on a copy with ONE LETTER changed in Bereishit 1:1 and integrity.sha256 untouched — L1_INTEGRITY_FAILED, l1_hash fail, cloud not readable, /peah/ask 503. Not done: the Haskell side. Luchot.Firmware.l1ExpectedHash is compared with nothing; the engine reads L1 without hashing it (that would need a SHA-256 dependency). The engine is reached only through the pipeline, which now gates every cycle on the real check. Co-Authored-By: Claude Fable 5.1 --- KOHEN.md | 8 +- KOHEN_PROMPT.md | 3 +- deploy/ARCHITECTURE.md | 4 +- scripts/verify_l1.py | 59 +++++ specs/shorashim_architecture.md | 17 +- specs/shorashim_vayakhel.md | 4 +- vav_pipeline/lib/vav_pipeline/application.ex | 3 + vav_pipeline/lib/vav_pipeline/doorpost.ex | 2 +- vav_pipeline/lib/vav_pipeline/l1_integrity.ex | 155 +++++++++++ vav_pipeline/lib/vav_pipeline/l1_reader.ex | 94 ++++++- .../test/tablet38_l1_integrity_test.exs | 247 ++++++++++++++++++ 11 files changed, 579 insertions(+), 17 deletions(-) create mode 100755 scripts/verify_l1.py create mode 100644 vav_pipeline/lib/vav_pipeline/l1_integrity.ex create mode 100644 vav_pipeline/test/tablet38_l1_integrity_test.exs diff --git a/KOHEN.md b/KOHEN.md index ab07276f..5db8e7a4 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -102,10 +102,12 @@ Confirm L1 is in the repository and the integrity hash matches the firmware cons ```bash cd ~/shorashim # or wherever you cloned -# 1. The hash file matches the constant baked into the binaries +# 1. The scroll itself hashes to the constant baked into the binaries. +# (Until 2026-10-04 this step compared integrity.sha256 with the +# constant — two copies of one string. This reads every verse.) expected="156b1d2724a4bf9999aa45c83ebd0794792f5aa4d524e8a7a898c871c1d8c9d5" -actual=$(cat l1/integrity.sha256) -[ "$expected" = "$actual" ] && echo "OK: hash matches" || echo "FAIL: hash mismatch" +python3 scripts/verify_l1.py l1 "$expected" +# Expect: computed = recorded = expected, "OK: the scroll matches" # 2. Verses are present verse_count=$(find l1/torah -name "*.json" -not -name "index.json" | wc -l) diff --git a/KOHEN_PROMPT.md b/KOHEN_PROMPT.md index eb801c90..b8f80407 100644 --- a/KOHEN_PROMPT.md +++ b/KOHEN_PROMPT.md @@ -78,8 +78,7 @@ test -d ~/shorashim/specs && test -d ~/shorashim/l1/torah && echo "repo OK" # 0d. L1 integrity matches the firmware constant cd ~/shorashim expected="156b1d2724a4bf9999aa45c83ebd0794792f5aa4d524e8a7a898c871c1d8c9d5" -actual=$(cat l1/integrity.sha256) -[ "$expected" = "$actual" ] && echo "L1 OK" || echo "L1 FAIL — STOP" +python3 scripts/verify_l1.py l1 "$expected" || echo "L1 FAIL — STOP" # 0e. Specification identity (Doorpost) cd ~/shorashim/vav_pipeline diff --git a/deploy/ARCHITECTURE.md b/deploy/ARCHITECTURE.md index 737710f0..04fd74e0 100644 --- a/deploy/ARCHITECTURE.md +++ b/deploy/ARCHITECTURE.md @@ -20,7 +20,7 @@ All four run in the same pod. vav-pipeline is the main container; talmudic-engin | Dependency | Purpose | Contract | |-----------|---------|----------| | Managed PostgreSQL 17+ | L3 Oral Tradition (append-only) | `DATABASE_URL` env var. Requires SSL. Role with `SELECT, INSERT` only — `UPDATE`, `DELETE` revoked (see migration). | -| (none — in the images) | L1 Torah filesystem (~29 MB) | Baked into every image at `/mnt/torah`, owned by root with no write bit. No volume. The recorded SHA-256 is compared with the pinned hash before every cycle. | +| (none — in the images) | L1 Torah filesystem (~29 MB) | Baked into every image at `/mnt/torah`, owned by root with no write bit. No volume. The pipeline hashes the tree at startup and every ten minutes; no cycle runs on a scroll that does not match the pinned hash. | | Block-storage PersistentVolume | Manna provision and its ledger (read-write, 1 GiB) | Mount at `/var/lib/shorashim/manna`. Operator writes `today.json` daily; the vessel writes `ledger.json`. | | Ingress controller + TLS cert | Public HTTPS access | Cert-manager with Let's Encrypt HTTP-01 is the reference setup. | | Container registry | Image distribution | Any registry that supports Docker v2 protocol. | @@ -54,7 +54,7 @@ Four Dockerfiles in `deploy/docker/`: The system reads three things from its environment: -1. **L1 Torah** — in the image at `$L1_MOUNT_POINT`, not writable by the service user. Its recorded hash is checked before every cycle. +1. **L1 Torah** — in the image at `$L1_MOUNT_POINT`, not writable by the service user. Read and hashed at startup and every ten minutes; every cycle is gated on the last reading. 2. **Manna provision** — JSON file at `$MANNA_PROVISION_FILE`, written by the kohen/treasurer. 3. **L3 database** — connection string in `$DATABASE_URL`. diff --git a/scripts/verify_l1.py b/scripts/verify_l1.py new file mode 100755 index 00000000..2e8d7738 --- /dev/null +++ b/scripts/verify_l1.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +""" +Read the scroll and say whether it is the scroll that was engraved. + +Hashes every file under l1/torah — path, then bytes, in the order +build_l1.py hashed them — and compares the result with the hash recorded +in l1/integrity.sha256 (and with an expected hash, if one is given). + +Comparing integrity.sha256 with a constant proves only that the two +copies of the constant agree. This reads the verses. + +Usage: + scripts/verify_l1.py # l1/ beside this script's repo + scripts/verify_l1.py /mnt/torah # another L1 root + scripts/verify_l1.py l1 # also require this exact hash + +Exit status 0 if everything agrees, 1 if anything does not. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from build_l1 import generate_integrity_hash # noqa: E402 + + +def main(argv: list[str]) -> int: + repo_root = Path(__file__).resolve().parent.parent + l1_root = Path(argv[1]) if len(argv) > 1 else repo_root / "l1" + expected = argv[2].strip().lower() if len(argv) > 2 else None + + torah = l1_root / "torah" + if not torah.is_dir(): + print(f"FAIL: {torah} is not a directory") + return 1 + + computed = generate_integrity_hash(str(torah)) + print(f"computed {computed}") + + ok = True + record = l1_root / "integrity.sha256" + if record.is_file(): + recorded = record.read_text().strip() + print(f"recorded {recorded}") + ok = ok and recorded == computed + else: + print(f"recorded (no {record})") + ok = False + + if expected is not None: + print(f"expected {expected}") + ok = ok and expected == computed + + print("OK: the scroll matches" if ok else "FAIL: the scroll does not match") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/specs/shorashim_architecture.md b/specs/shorashim_architecture.md index e1e593fe..3d6cb0ff 100644 --- a/specs/shorashim_architecture.md +++ b/specs/shorashim_architecture.md @@ -79,16 +79,23 @@ Exact versions are fixed in Construction §1.1; these are the ones built. | Layer | Technology | Purpose | |-------|-----------|---------| | L0 — Luchot | Compiled Haskell constants (`Luchot.Firmware`) | Immutable firmware baked into binary | -| L1 — Torah scroll | Baked into every image at `/mnt/torah`, owned by root with no write bit; its recorded SHA-256 is compared with the pinned hash before every cycle | Fixed specification | +| L1 — Torah scroll | Baked into every image at `/mnt/torah`, owned by root with no write bit; read and SHA-256-hashed at startup and every ten minutes, and no cycle runs on a scroll that does not match its pinned hash | Fixed specification | | L2 — Urim v'Tumim | A process in the Vav Pipeline (`UrimVTumim`). Not yet exposed over HTTP and not yet called by the cycle (Step 24) | Decision oracle interface | | L3 — Oral tradition | PostgreSQL 17, managed; append-only by `REVOKE UPDATE, DELETE` from the application role | Accumulated learning | L1 is read-only by ownership, not by mount: the service user can read the scroll and cannot change it. It is not a squashfs image and the -container's root filesystem is not mounted read-only. The integrity check -compares the hash recorded in `integrity.sha256` with the hash compiled -into the pipeline; it does not re-hash the verse files, so it witnesses -which L1 was built into the image, not that the files are unaltered since. +container's root filesystem is not mounted read-only. + +Its integrity is witnessed by reading it. A process in the pipeline +(`L1Integrity`) hashes every file under `torah/` — path and bytes, in the +order the builder hashed them — when the pipeline starts and again every +ten minutes. Every cycle and every health check is answered from the last +reading: if the scroll does not hash to the value compiled into the +pipeline, the cloud is not visible, no cycle runs, and the vessel reports +itself not ready. A change is therefore noticed within ten minutes, not +at the instant it is made. `scripts/verify_l1.py` does the same reading by +hand. ### 2.3 Infrastructure diff --git a/specs/shorashim_vayakhel.md b/specs/shorashim_vayakhel.md index 2343082d..809dbe19 100644 --- a/specs/shorashim_vayakhel.md +++ b/specs/shorashim_vayakhel.md @@ -43,7 +43,7 @@ The specification that sits inside the aron must be specified with the same prec | **Digital edition** | Westminster Leningrad Codex (WLC) 4.22 | | **Source** | Sefaria Export (text, cantillation, vowels) cross-validated against TANACH.us WLC data | | **Format** | JSON per verse: `{ "ref": "Genesis 1:1", "hebrew": "...", "consonantal": "...", "cantillation": [...], "ketiv_qere": [...] }` | -| **Integrity** | SHA-256 hash of complete L1 filesystem image, recorded at build time and verified at every mount | +| **Integrity** | SHA-256 over every file of the L1 tree (path, then bytes), recorded at build time in `integrity.sha256`. The pipeline re-reads and re-hashes the tree at startup and every ten minutes | | **Cantillation marks** | Unicode cantillation codepoints (U+0591–U+05AF) preserved as raw marks, NOT pre-parsed | | **Ketiv/qere** | Derived from WLC ketiv/qere annotations; stored as `{ "position": [book, ch, vs, word], "ketiv": "...", "qere": "..." }` | | **Gematria index** | Computed once from the consonantal text at build time, stored as lookup table, verified against independent computation | @@ -59,7 +59,7 @@ The specification that sits inside the aron must be specified with the same prec 7. Verify SHA-256 matches at every system boot 8. Record the hash in L0 (compiled Haskell constant) — the luchot carry the fingerprint of the scroll -If the hash does not match, the system does not start. The specification's integrity is verified before the system opens its eyes. +If the scroll does not hash to the pinned value, the system does not serve: no cycle runs and the vessel reports itself not ready. The specification's integrity is verified before the system opens its eyes, and again every ten minutes while they are open. ### 1.3 Haskell Dependencies (Exact) diff --git a/vav_pipeline/lib/vav_pipeline/application.ex b/vav_pipeline/lib/vav_pipeline/application.ex index 6f21b29c..2518d635 100644 --- a/vav_pipeline/lib/vav_pipeline/application.ex +++ b/vav_pipeline/lib/vav_pipeline/application.ex @@ -33,6 +33,9 @@ defmodule VavPipeline.Application do children = [ {VavPipeline.Chatzotzrot, []}, + # Tablet 38 — reads the scroll and hashes it; everything that asks + # "is L1 intact?" is answered from this reading. + {VavPipeline.L1Integrity, []}, {VavPipeline.AreiMiklat, []}, {VavPipeline.EglahArufah, []}, {VavPipeline.Bikurim, []}, diff --git a/vav_pipeline/lib/vav_pipeline/doorpost.ex b/vav_pipeline/lib/vav_pipeline/doorpost.ex index cc7b0427..f83b67be 100644 --- a/vav_pipeline/lib/vav_pipeline/doorpost.ex +++ b/vav_pipeline/lib/vav_pipeline/doorpost.ex @@ -175,7 +175,7 @@ defmodule VavPipeline.Doorpost do # changes; the update is a deliberate operator commit. A bare # hash-update commit (without a corresponding spec change in the # same commit) is itself a drift signal. - @expected_hash "7e5286b99aaefa49c7516d3e27dae9d7e3f4fc9ae89347c92c5394685d953400" + @expected_hash "3f6d47a0b91ce12a1002cfe2d0a6bfe7af4fb280cd05264c87ba914d258a06fe" @doc "Return the list of files that constitute the category-grade specification." @spec files() :: [String.t()] diff --git a/vav_pipeline/lib/vav_pipeline/l1_integrity.ex b/vav_pipeline/lib/vav_pipeline/l1_integrity.ex new file mode 100644 index 00000000..8e5b618c --- /dev/null +++ b/vav_pipeline/lib/vav_pipeline/l1_integrity.ex @@ -0,0 +1,155 @@ +defmodule VavPipeline.L1Integrity do + @moduledoc """ + The witness that the scroll in this vessel is the scroll that was engraved. + + Broken tablet 38 (2026-10-04). `L1Reader.verify_integrity/0` read the + hash stored in `integrity.sha256` and compared that string with the hash + compiled into this module's neighbour. It never opened a verse. Two + copies of one constant were compared with each other, and on that + comparison rested `l1_hash: pass`, the Cloud's "L1 corrupted — the cloud + is not visible", and Construction §1.2's "if the hash does not match, the + system does not start." A verse could have been changed and every one of + them would have gone on saying *tov*. + + This process reads the scroll. When it starts it hashes the whole of + `torah/` — every path and every byte, in the order `scripts/build_l1.py` + hashed them when L1 was engraved — and it does so again at an interval + (ten minutes unless configured). `L1Reader.verify_integrity/0` asks this + process what it found; the answer is a comparison of what is on disk + with what was pinned, not of the pin with itself. + + Hashing 5,859 files takes a fraction of a second, which is too long to + do before every cycle and every health probe; so the cycle is answered + from the last reading, and the reading is never older than the interval. + Callers that arrive while the first reading is under way wait for it. + + *Lo tosif alav v'lo tigra mimenu* — you shall not add to it and you shall + not take from it (Devarim 13:1). The check is how the vessel knows. + """ + + use GenServer + require Logger + + alias VavPipeline.L1Reader + + @default_interval :timer.minutes(10) + + @type reading :: %{ + status: :ok | {:error, String.t()}, + computed: String.t() | nil, + expected: String.t(), + read_at: DateTime.t() + } + + def start_link(opts \\ []) do + GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name, __MODULE__)) + end + + @doc "`:ok` if the scroll on disk hashes to the pinned hash, else `{:error, reason}`." + @spec status(GenServer.server()) :: :ok | {:error, String.t()} + def status(server \\ __MODULE__), do: GenServer.call(server, :status, 60_000) + + @doc "The last reading in full — what was computed, what was expected, and when." + @spec reading(GenServer.server()) :: reading() + def reading(server \\ __MODULE__), do: GenServer.call(server, :reading, 60_000) + + @doc "Read the scroll again now, and return what was found." + @spec recheck(GenServer.server()) :: :ok | {:error, String.t()} + def recheck(server \\ __MODULE__), do: GenServer.call(server, :recheck, 60_000) + + # ------------------------------------------------------------ + # GenServer + # ------------------------------------------------------------ + + @impl true + def init(opts) do + state = %{ + root: Keyword.get(opts, :l1_root, L1Reader.l1_root()), + expected: Keyword.get(opts, :expected_hash, L1Reader.expected_hash()), + interval: + Keyword.get( + opts, + :interval, + Application.get_env(:vav_pipeline, :l1_rehash_interval_ms, @default_interval) + ), + reading: nil + } + + {:ok, state, {:continue, :read}} + end + + @impl true + def handle_continue(:read, state) do + state = read(state) + schedule(state) + {:noreply, state} + end + + @impl true + def handle_info(:read, state) do + state = read(state) + schedule(state) + {:noreply, state} + end + + @impl true + def handle_call(:status, _from, state), do: {:reply, state.reading.status, state} + def handle_call(:reading, _from, state), do: {:reply, state.reading, state} + + def handle_call(:recheck, _from, state) do + state = read(state) + {:reply, state.reading.status, state} + end + + defp schedule(%{interval: :never}), do: :ok + defp schedule(%{interval: ms}), do: Process.send_after(self(), :read, ms) + + defp read(state) do + {micros, result} = :timer.tc(fn -> L1Reader.compute_tree_hash(state.root) end) + previous = state.reading && state.reading.status + + {status, computed} = + case result do + {:ok, hash} when hash == state.expected -> + {:ok, hash} + + {:ok, hash} -> + {{:error, + "L1_INTEGRITY_FAILED: the scroll on disk hashes to #{hash}, " <> + "expected #{state.expected}"}, hash} + + {:error, reason} -> + {{:error, "L1_INTEGRITY_FAILED: the scroll could not be read — #{reason}"}, nil} + end + + announce(status, previous, div(micros, 1000)) + + %{ + state + | reading: %{ + status: status, + computed: computed, + expected: state.expected, + read_at: DateTime.utc_now() + } + } + end + + # Say it when it is first known and whenever it changes; stay quiet on + # the ten-minute readings that find what the last one found. + defp announce(status, status, _ms), do: :ok + + defp announce(:ok, _previous, ms) do + Logger.info("L1Integrity: the scroll was read and matches its pinned hash (#{ms}ms)") + end + + defp announce({:error, reason}, _previous, _ms) do + Logger.error("L1Integrity: #{reason}") + + if Process.whereis(VavPipeline.Chatzotzrot) do + VavPipeline.Chatzotzrot.signal(:assembly_leaders, reason: reason) + end + + :ok + end +end diff --git a/vav_pipeline/lib/vav_pipeline/l1_reader.ex b/vav_pipeline/lib/vav_pipeline/l1_reader.ex index a6f17e6e..bc049797 100644 --- a/vav_pipeline/lib/vav_pipeline/l1_reader.ex +++ b/vav_pipeline/lib/vav_pipeline/l1_reader.ex @@ -244,10 +244,22 @@ defmodule VavPipeline.L1Reader do end @doc """ - Verify L1 integrity by checking the SHA-256 hash. + Verify L1 integrity: the scroll on disk must hash to the pinned hash. This is a critical check — if L1 is corrupted, the pipeline must halt. The specification store is inviolable. + + Two things are asked, and both must hold: + + 1. The hash recorded beside the scroll (`integrity.sha256`) is the + pinned one — this L1 is the L1 this build was made for. + 2. The scroll itself, read from disk and hashed, gives that hash + (`VavPipeline.L1Integrity`, which reads it at startup and again + every ten minutes). + + Until tablet 38 only the first was asked, and the first compares a + recorded string with a compiled-in string: it could not notice a + changed verse. """ @spec verify_integrity() :: :ok | {:error, String.t()} def verify_integrity do @@ -258,7 +270,7 @@ defmodule VavPipeline.L1Reader do stored_hash = contents |> String.trim() if stored_hash == @l1_expected_hash do - :ok + verify_scroll() else {:error, "L1_INTEGRITY_FAILED: hash mismatch — stored #{stored_hash}, expected #{@l1_expected_hash}"} end @@ -268,6 +280,84 @@ defmodule VavPipeline.L1Reader do end end + # What the scroll on disk actually hashes to. The supervised witness + # holds a recent reading; where it is not running (a bare unit test, + # a one-off `eval`), the scroll is read here and now. + defp verify_scroll do + if Process.whereis(VavPipeline.L1Integrity) do + VavPipeline.L1Integrity.status() + else + case compute_tree_hash() do + {:ok, @l1_expected_hash} -> + :ok + + {:ok, other} -> + {:error, + "L1_INTEGRITY_FAILED: the scroll on disk hashes to #{other}, expected #{@l1_expected_hash}"} + + {:error, reason} -> + {:error, "L1_INTEGRITY_FAILED: the scroll could not be read — #{reason}"} + end + end + end + + @doc """ + Hash the scroll: SHA-256 over every file under `/torah`, each + contributing its path relative to `torah/` and then its bytes. + + The order is the order `scripts/build_l1.py` (`generate_integrity_hash`) + used when L1 was engraved, and must stay so or the two will disagree: + directories in ascending order of their path, and within a directory the + files in ascending order of name — plain string order, so `10.json` + comes before `2.json`. + """ + @spec compute_tree_hash(String.t()) :: {:ok, String.t()} | {:error, String.t()} + def compute_tree_hash(root \\ l1_root()) do + torah = Path.join(root, "torah") + + if File.dir?(torah) do + digest = + [torah | subdirectories(torah)] + |> Enum.sort() + |> Enum.reduce(:crypto.hash_init(:sha256), fn dir, ctx -> + dir + |> File.ls!() + |> Enum.sort() + |> Enum.reduce(ctx, fn name, ctx -> + path = Path.join(dir, name) + + if File.regular?(path) do + ctx + |> :crypto.hash_update(Path.relative_to(path, torah)) + |> :crypto.hash_update(File.read!(path)) + else + ctx + end + end) + end) + |> :crypto.hash_final() + |> Base.encode16(case: :lower) + + {:ok, digest} + else + {:error, "#{torah} is not a directory"} + end + rescue + e in [File.Error] -> {:error, Exception.message(e)} + end + + defp subdirectories(dir) do + dir + |> File.ls!() + |> Enum.map(&Path.join(dir, &1)) + |> Enum.filter(&File.dir?/1) + |> Enum.flat_map(fn sub -> [sub | subdirectories(sub)] end) + end + + @doc "The pinned L1 hash this build expects." + @spec expected_hash() :: String.t() + def expected_hash, do: @l1_expected_hash + @doc """ Get the root path of L1. """ diff --git a/vav_pipeline/test/tablet38_l1_integrity_test.exs b/vav_pipeline/test/tablet38_l1_integrity_test.exs new file mode 100644 index 00000000..d5ff2ca2 --- /dev/null +++ b/vav_pipeline/test/tablet38_l1_integrity_test.exs @@ -0,0 +1,247 @@ +defmodule VavPipeline.Tablet38L1IntegrityTest do + @moduledoc """ + Tablet 38 (2026-10-04) — the integrity check did not check integrity. + + `L1Reader.verify_integrity/0` compared the hash recorded in + `integrity.sha256` with the hash compiled into the module. It never read + a verse. Every test of it passed, because every test asked it about an + untouched L1 — the one question a check that checks nothing answers + correctly. + + These tests change the scroll and ask again. + """ + + use ExUnit.Case, async: false + + import Plug.Test + import Plug.Conn + + alias VavPipeline.{HaltMonitor, L1Integrity, L1Reader, MannaLedger} + alias VavPipeline.BigdeiKodesh.Router + + @opts Router.init([]) + + # ============================================================ + # The hash itself + # ============================================================ + describe "compute_tree_hash/1" do + test "of the real scroll is the pinned hash, and the hash build_l1.py recorded" do + assert {:ok, hash} = L1Reader.compute_tree_hash() + assert hash == L1Reader.expected_hash() + + recorded = L1Reader.l1_root() |> Path.join("integrity.sha256") |> File.read!() |> String.trim() + assert hash == recorded + end + + test "hashes path then bytes, directories then files, in plain string order" do + root = small_scroll() + + # The order build_l1.py walks: the torah root's own files, then each + # directory by path; within a directory, names as strings — so + # 10.json comes before 2.json. + expected = + [ + {"index.json", "INDEX"}, + {"alef/1/1.json", "one"}, + {"alef/1/10.json", "ten"}, + {"alef/1/2.json", "two"}, + {"alef/2/1.json", "second chapter"} + ] + |> Enum.reduce(:crypto.hash_init(:sha256), fn {rel, bytes}, ctx -> + ctx |> :crypto.hash_update(rel) |> :crypto.hash_update(bytes) + end) + |> :crypto.hash_final() + |> Base.encode16(case: :lower) + + assert {:ok, ^expected} = L1Reader.compute_tree_hash(root) + end + + test "one changed byte, one added file, one renamed file — each changes the hash" do + root = small_scroll() + {:ok, original} = L1Reader.compute_tree_hash(root) + verse = Path.join([root, "torah", "alef", "1", "2.json"]) + + File.write!(verse, "twO") + assert {:ok, changed} = L1Reader.compute_tree_hash(root) + assert changed != original + + File.write!(verse, "two") + assert {:ok, ^original} = L1Reader.compute_tree_hash(root) + + File.write!(Path.join([root, "torah", "alef", "1", "3.json"]), "an added verse") + assert {:ok, added} = L1Reader.compute_tree_hash(root) + assert added != original + File.rm!(Path.join([root, "torah", "alef", "1", "3.json"])) + + File.rename!(verse, Path.join([root, "torah", "alef", "1", "20.json"])) + assert {:ok, renamed} = L1Reader.compute_tree_hash(root) + assert renamed != original + end + + test "a scroll that is not there is an error, not a hash" do + assert {:error, reason} = L1Reader.compute_tree_hash("/nonexistent/l1") + assert reason =~ "not a directory" + end + end + + # ============================================================ + # The witness + # ============================================================ + describe "L1Integrity" do + test "reads the scroll when it starts, and again when asked" do + root = small_scroll() + {:ok, pinned} = L1Reader.compute_tree_hash(root) + + {:ok, pid} = + L1Integrity.start_link(name: :tablet38_witness, l1_root: root, expected_hash: pinned, interval: :never) + + on_exit(fn -> if Process.alive?(pid), do: GenServer.stop(pid) end) + + assert :ok = L1Integrity.status(:tablet38_witness) + assert %{computed: ^pinned, expected: ^pinned} = L1Integrity.reading(:tablet38_witness) + + # Someone alters a verse. The last reading still says tov — it is a + # reading, not a guard — until the scroll is read again. + File.write!(Path.join([root, "torah", "alef", "1", "1.json"]), "0ne") + assert {:error, reason} = L1Integrity.recheck(:tablet38_witness) + assert reason =~ "L1_INTEGRITY_FAILED" + assert reason =~ "expected #{pinned}" + assert {:error, _} = L1Integrity.status(:tablet38_witness) + + # Restored, it is tov again. + File.write!(Path.join([root, "torah", "alef", "1", "1.json"]), "one") + assert :ok = L1Integrity.recheck(:tablet38_witness) + end + + test "re-reads on its own at the interval" do + root = small_scroll() + {:ok, pinned} = L1Reader.compute_tree_hash(root) + + {:ok, pid} = + L1Integrity.start_link(name: :tablet38_periodic, l1_root: root, expected_hash: pinned, interval: 50) + + on_exit(fn -> if Process.alive?(pid), do: GenServer.stop(pid) end) + + assert :ok = L1Integrity.status(:tablet38_periodic) + File.write!(Path.join([root, "torah", "index.json"]), "INDEX, altered") + + assert eventually(fn -> match?({:error, _}, L1Integrity.status(:tablet38_periodic)) end) + end + end + + # ============================================================ + # The whole path: an altered scroll stops the vessel + # ============================================================ + describe "with a scroll that does not match its pin" do + setup do + if HaltMonitor.halted?(), do: HaltMonitor.resolve("tablet38 test setup") + MannaLedger.reset() + write_provision(20) + + # Sanity: with the real scroll, all is well. + assert :ok = L1Reader.verify_integrity() + + # Replace the supervised witness with one that reads an altered + # copy — the recorded hash file is untouched and still "matches", + # which is exactly the case the old check could not see. + root = small_scroll() + :ok = Supervisor.terminate_child(VavPipeline.Supervisor, L1Integrity) + + {:ok, pid} = + L1Integrity.start_link(l1_root: root, expected_hash: L1Reader.expected_hash(), interval: :never) + + on_exit(fn -> + if Process.alive?(pid), do: GenServer.stop(pid) + {:ok, _} = Supervisor.restart_child(VavPipeline.Supervisor, L1Integrity) + MannaLedger.reset() + File.rm(VavPipeline.Manna.provision_file()) + end) + + :ok + end + + test "verify_integrity fails, though integrity.sha256 still holds the pinned hash" do + recorded = L1Reader.l1_root() |> Path.join("integrity.sha256") |> File.read!() |> String.trim() + assert recorded == L1Reader.expected_hash() + + assert {:error, reason} = L1Reader.verify_integrity() + assert reason =~ "L1_INTEGRITY_FAILED" + assert reason =~ "the scroll on disk hashes to" + end + + test "health says l1_hash: fail and the vessel is not ready" do + conn = conn(:get, "/kodesh/health") |> Router.call(@opts) + body = Jason.decode!(conn.resp_body) + + assert conn.status == 503 + assert body["checks"]["l1_hash"] == "fail" + end + + test "no cycle runs on it — /peah/ask does not answer from an altered scroll" do + conn = + conn(:post, "/peah/ask", Jason.encode!(%{question: "בראשית ברא אלהים"})) + |> put_req_header("content-type", "application/json") + |> Router.call(@opts) + + refute conn.status == 200 + refute Map.has_key?(Jason.decode!(conn.resp_body), "answer") + end + end + + # ------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------ + + defp small_scroll do + root = Path.join(System.tmp_dir!(), "tablet38_#{System.unique_integer([:positive])}") + on_exit(fn -> File.rm_rf!(root) end) + + files = %{ + "index.json" => "INDEX", + "alef/1/1.json" => "one", + "alef/1/2.json" => "two", + "alef/1/10.json" => "ten", + "alef/2/1.json" => "second chapter" + } + + for {rel, bytes} <- files do + path = Path.join([root, "torah", rel]) + File.mkdir_p!(Path.dirname(path)) + File.write!(path, bytes) + end + + root + end + + defp eventually(fun, tries \\ 100) do + cond do + fun.() -> true + tries == 0 -> false + true -> Process.sleep(20) && eventually(fun, tries - 1) + end + end + + defp write_provision(compute_units) do + today = Date.utc_today() |> Date.to_iso8601() + expires = DateTime.utc_now() |> DateTime.add(6 * 3600) |> DateTime.to_iso8601() + + provision = %{ + "date" => today, + "provision" => %{ + "compute_units" => compute_units, + "memory_mb" => 512, + "api_calls_external" => 5, + "storage_delta_mb" => 2, + "sanhedrin_panels_max" => 2 + }, + "type" => "regular", + "expires_at" => expires, + "written_by" => "tablet38_test", + "written_at" => DateTime.utc_now() |> DateTime.to_iso8601() + } + + path = VavPipeline.Manna.provision_file() + File.mkdir_p!(Path.dirname(path)) + File.write!(path, Jason.encode!(provision)) + end +end