L1Integrity: the ten-minute re-reading runs beside the witness, not in it

On the production node a reading of the scroll takes about three
seconds (3310 ms at first start; 0.9 s on the build host). The periodic
re-reading ran inside the GenServer, so once every ten minutes every
cycle and every readiness probe that asked "is L1 intact?" would have
waited those seconds behind it.

The periodic reading is now done by a task that reports back; the
process goes on answering from its last reading meanwhile. The first
reading is still made in the process itself — nobody is answered before
the scroll has been read once — and an explicit recheck/0 is still
synchronous. The next reading is scheduled when one begins, not when it
reports, so a reading that dies silently is not the last.

Two tests: it re-reads on its own in both directions (altered → fail,
restored → tov), and a status call during a periodic reading returns at
once. 484 tests, 0 failures on seeds 0 7 42 99.

Built image, run with a live-mounted scroll: l1_hash pass, /peah/ask
200; then אחד in Devarim 6:4 was changed to אחר while the vessel ran
and the witness asked to read again — L1_INTEGRITY_FAILED, l1_hash
fail, /peah/ask 503.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

vav_pipeline/lib/vav_pipeline/l1_integrity.ex

@@ -18,10 +18,14 @@ defmodule VavPipeline.L1Integrity do
   process what it found; the answer is a comparison of what is on disk
   with what was pinned, not of the pin with itself.
 
-  Hashing 5,859 files takes a fraction of a second, which is too long to
-  do before every cycle and every health probe; so the cycle is answered
-  from the last reading, and the reading is never older than the interval.
-  Callers that arrive while the first reading is under way wait for it.
+  Hashing 5,859 files takes about a second (three on the production
+  node), which is too long to do before every cycle and every health
+  probe; so the cycle is answered from the last reading, and the reading
+  is never older than the interval. Callers that arrive while the FIRST
+  reading is under way wait for it — nobody is answered before the scroll
+  has been read once. The periodic re-readings run beside the process,
+  not in it: while one is under way the last reading goes on answering,
+  and no cycle or probe waits three seconds for it.
 
   *Lo tosif alav v'lo tigra mimenu* — you shall not add to it and you shall
   not take from it (Devarim 13:1). The check is how the vessel knows.
@@ -85,13 +89,27 @@ defmodule VavPipeline.L1Integrity do
     {:noreply, state}
   end
 
+  # The periodic reading is done by a task, so that this process stays
+  # free to answer from the last one.
   @impl true
   def handle_info(:read, state) do
-    state = read(state)
+    witness = self()
+    root = state.root
+
+    Task.start(fn ->
+      send(witness, {:read_done, :timer.tc(fn -> L1Reader.compute_tree_hash(root) end)})
+    end)
+
+    # The next reading is scheduled now, not when this one reports back:
+    # a reading that died without a word must not be the last one.
     schedule(state)
     {:noreply, state}
   end
 
+  def handle_info({:read_done, {micros, result}}, state) do
+    {:noreply, record(state, result, micros)}
+  end
+
   @impl true
   def handle_call(:status, _from, state), do: {:reply, state.reading.status, state}
   def handle_call(:reading, _from, state), do: {:reply, state.reading, state}
@@ -106,6 +124,10 @@ defmodule VavPipeline.L1Integrity do
 
   defp read(state) do
     {micros, result} = :timer.tc(fn -> L1Reader.compute_tree_hash(state.root) end)
+    record(state, result, micros)
+  end
+
+  defp record(state, result, micros) do
     previous = state.reading && state.reading.status
 
     {status, computed} =

vav_pipeline/test/tablet38_l1_integrity_test.exs

@@ -126,6 +126,27 @@ defmodule VavPipeline.Tablet38L1IntegrityTest do
       File.write!(Path.join([root, "torah", "index.json"]), "INDEX, altered")
 
       assert eventually(fn -> match?({:error, _}, L1Integrity.status(:tablet38_periodic)) end)
+
+      # And it keeps re-reading: restored, it is tov again without being asked.
+      File.write!(Path.join([root, "torah", "index.json"]), "INDEX")
+      assert eventually(fn -> L1Integrity.status(:tablet38_periodic) == :ok end)
+    end
+
+    test "answers from the last reading while a periodic re-reading is under way" do
+      root = small_scroll()
+      {:ok, pinned} = L1Reader.compute_tree_hash(root)
+
+      {:ok, pid} =
+        L1Integrity.start_link(name: :tablet38_busy, l1_root: root, expected_hash: pinned, interval: :never)
+
+      on_exit(fn -> if Process.alive?(pid), do: GenServer.stop(pid) end)
+
+      # Trigger a periodic reading by hand; the process must not be busy
+      # with it — a status call is answered at once, from the last reading.
+      send(pid, :read)
+      {micros, status} = :timer.tc(fn -> L1Integrity.status(:tablet38_busy) end)
+      assert status == :ok
+      assert micros < 50_000
     end
   end