probes: liveness asks "is it dead?", readiness asks "may it serve?" (kohen Day 3, restart 1)

TORAH ANCHOR — Shemot 16:4, and tamid §2.1 check 3:
  Manna provision read … If absent: WAIT. Alert senior kohanim.
WAIT. Not die. Provision is not the health of the vessel. Killing the
process does not produce manna; it re-runs init, re-spawns the Port,
and waits again — three minutes later.

WHAT THE KOHEN FOUND (BUILDER_HANDOFF_probes.md, 2026-09-07 20:35Z):

A fresh pod after recreate. emptyDir manna empty — by design
(KOHEN.md:406). manna_present honestly read "fail" — the Day-6 fix,
correct per tamid §1.1. overall → "degraded" → /kodesh/health 503.
BOTH probes pointed at /kodesh/health. Liveness: period 60s, default
failureThreshold 3 → after three minutes the kubelet SIGTERMed a
process whose log showed the Port alive, zero errors, l3_writable
pass, esh_tamid pass. It was alive and serving. It was killed for
lacking provision, and would be again every three minutes until the
kohen wrote today.json — i.e. on EVERY pod creation.

Introduced by a correct fix (f1ab7c8 made health honest); the missing
piece was the distinction between "not ready to serve" and "dead".

WHAT CHANGED:

bigdei_kodesh/router.ex
  GET /kodesh/live — 200 iff the BEAM answers HTTP and the HaltMonitor
  process is registered. Looks at NOTHING that is a wait state: no
  manna, no L3, no cloud, no probe cycle. And deliberately NOT "not
  halted": a HALTED system is a chosen state (a cardinal sin was
  detected); restarting it would clear the halt and lose the event.
  Liveness measures that the monitor is ALIVE, not that it is quiet.
  /kodesh/health is unchanged — it is readiness, and a vessel without
  provision correctly does not receive offerings (that is shema; the
  Service stops routing to it).

deploy/k8s/deployment.yaml
  livenessProbe → /kodesh/live, failureThreshold 3 made explicit.
  readinessProbe stays on /kodesh/health.

test/liveness_test.exs — 3 tests
  live 200 with NO manna while health is 503; live 200 while HALTED
  (and pe'ah answers 503 HALT_*, not a served result); live touches
  neither the ledger nor L3 nor the cycle.

KOHEN.md §Day 3c — a fresh pod reads 2/3 Ready, 0 restarts until Day 4
writes manna, and that is correct; restarts on a pod whose log shows
the Port alive and zero errors mean liveness is on the wrong endpoint.

ALSO — a leak of my own from f1ab7c8:
  The MannaLedger is a global GenServer keyed by date. The Day-6
  "provision exhausted" test left it at allocated 1 / consumed 1 for
  today; when step31 Test 4 later wrote a fresh today.json the ledger
  answered :already_loaded, consume failed, and the cycle waited —
  a 1-in-5-seeds flake. Fixed where it leaked (day6 + liveness tests
  reset the ledger on exit) and where it landed (step31 Test 4 gathers
  from a clean ledger, with the reason written down).

VERIFIED — the kohen's procedure, locally, DB deliberately dead:
  fresh, no manna     live 200   health 503 critical (manna + l3)
  manna written       live 200   health 503 degraded — l3_writable ONLY;
                                 manna_present flipped to pass
  manna removed       live 200   health 503
  exit/SIGTERM events in the container log: 0
  mix test — 440 tests, 0 failures, 36 excluded, across seeds 0 1 2 7 42
  kubectl apply --dry-run=client: deployment valid; liveness
    /kodesh/live (threshold 3), readiness /kodesh/health
  Doorpost.verify_identity() :ok — no @spec_files entry touched
  l1/integrity.sha256 unchanged: 99bbd07a…4ea377

RAISED, NOT FIXED: manna is an emptyDir, so every pod creation needs
the kohen to write it again. manna-volume.yaml's PVC exists for
exactly this and is "not applied at Day 3" by the runbook's own note.
That is an operations decision, not a probe one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

vav_pipeline/test/step31_shema_test.exs

@@ -316,6 +316,13 @@ defmodule VavPipeline.Step31.ShemaTest do
     end
 
     test "provision file present → system processes" do
+      # The cycle now CONSUMES from the MannaLedger (kohen Day 6). The
+      # ledger is a global GenServer keyed by date: a provision another
+      # test loaded and exhausted today would read :already_loaded and
+      # refuse the fresh file written below. Gather from a clean ledger.
+      VavPipeline.MannaLedger.reset()
+      on_exit(fn -> VavPipeline.MannaLedger.reset() end)
+
       # Write a valid provision file
       today = Date.utc_today() |> Date.to_iso8601()