probes: liveness asks "is it dead?", readiness asks "may it serve?" (kohen Day 3, restart 1)

TORAH ANCHOR — Shemot 16:4, and tamid §2.1 check 3:
  Manna provision read … If absent: WAIT. Alert senior kohanim.
WAIT. Not die. Provision is not the health of the vessel. Killing the
process does not produce manna; it re-runs init, re-spawns the Port,
and waits again — three minutes later.

WHAT THE KOHEN FOUND (BUILDER_HANDOFF_probes.md, 2026-09-07 20:35Z):

A fresh pod after recreate. emptyDir manna empty — by design
(KOHEN.md:406). manna_present honestly read "fail" — the Day-6 fix,
correct per tamid §1.1. overall → "degraded" → /kodesh/health 503.
BOTH probes pointed at /kodesh/health. Liveness: period 60s, default
failureThreshold 3 → after three minutes the kubelet SIGTERMed a
process whose log showed the Port alive, zero errors, l3_writable
pass, esh_tamid pass. It was alive and serving. It was killed for
lacking provision, and would be again every three minutes until the
kohen wrote today.json — i.e. on EVERY pod creation.

Introduced by a correct fix (f1ab7c8 made health honest); the missing
piece was the distinction between "not ready to serve" and "dead".

WHAT CHANGED:

bigdei_kodesh/router.ex
  GET /kodesh/live — 200 iff the BEAM answers HTTP and the HaltMonitor
  process is registered. Looks at NOTHING that is a wait state: no
  manna, no L3, no cloud, no probe cycle. And deliberately NOT "not
  halted": a HALTED system is a chosen state (a cardinal sin was
  detected); restarting it would clear the halt and lose the event.
  Liveness measures that the monitor is ALIVE, not that it is quiet.
  /kodesh/health is unchanged — it is readiness, and a vessel without
  provision correctly does not receive offerings (that is shema; the
  Service stops routing to it).

deploy/k8s/deployment.yaml
  livenessProbe → /kodesh/live, failureThreshold 3 made explicit.
  readinessProbe stays on /kodesh/health.

test/liveness_test.exs — 3 tests
  live 200 with NO manna while health is 503; live 200 while HALTED
  (and pe'ah answers 503 HALT_*, not a served result); live touches
  neither the ledger nor L3 nor the cycle.

KOHEN.md §Day 3c — a fresh pod reads 2/3 Ready, 0 restarts until Day 4
writes manna, and that is correct; restarts on a pod whose log shows
the Port alive and zero errors mean liveness is on the wrong endpoint.

ALSO — a leak of my own from f1ab7c8:
  The MannaLedger is a global GenServer keyed by date. The Day-6
  "provision exhausted" test left it at allocated 1 / consumed 1 for
  today; when step31 Test 4 later wrote a fresh today.json the ledger
  answered :already_loaded, consume failed, and the cycle waited —
  a 1-in-5-seeds flake. Fixed where it leaked (day6 + liveness tests
  reset the ledger on exit) and where it landed (step31 Test 4 gathers
  from a clean ledger, with the reason written down).

VERIFIED — the kohen's procedure, locally, DB deliberately dead:
  fresh, no manna     live 200   health 503 critical (manna + l3)
  manna written       live 200   health 503 degraded — l3_writable ONLY;
                                 manna_present flipped to pass
  manna removed       live 200   health 503
  exit/SIGTERM events in the container log: 0
  mix test — 440 tests, 0 failures, 36 excluded, across seeds 0 1 2 7 42
  kubectl apply --dry-run=client: deployment valid; liveness
    /kodesh/live (threshold 3), readiness /kodesh/health
  Doorpost.verify_identity() :ok — no @spec_files entry touched
  l1/integrity.sha256 unchanged: 99bbd07a…4ea377

RAISED, NOT FIXED: manna is an emptyDir, so every pod creation needs
the kohen to write it again. manna-volume.yaml's PVC exists for
exactly this and is "not applied at Day 3" by the runbook's own note.
That is an operations decision, not a probe one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex

@@ -161,12 +161,59 @@ defmodule VavPipeline.BigdeiKodesh.Router do
 
   # === KODESH — Operator Interface ===
 
+  # === KODESH — Readiness vs. liveness ===
+  #
+  # Two questions, two endpoints. The kohen's Day-3 (restart 1) finding:
+  # both probes pointed at /kodesh/health. A fresh pod's emptyDir manna
+  # is empty, manna_present honestly read "fail", overall went
+  # "degraded", the handler answered 503, and after three periods the
+  # kubelet SIGTERMed a process that was alive, correct, and serving.
+  # Every pod creation entered a 3-minute kill loop until manna landed.
+  #
+  # tamid §2.1 check 3: manna absent → WAIT. Not die. Provision is not
+  # the health of the vessel. Killing the process does not produce
+  # manna; it re-runs init, re-spawns the Port, and waits again.
+  #
+  #   /kodesh/health — READINESS. "May this vessel receive offerings?"
+  #     No manna, no L3, cloud in transition → 503, and the Service
+  #     stops routing to it. That is shema: a vessel without provision
+  #     does not serve. Correct, and unchanged.
+  #
+  #   /kodesh/live   — LIVENESS. "Must this process be restarted?"
+  #     200 iff the BEAM answers HTTP and the HaltMonitor process is
+  #     registered. NOT "not halted": a HALTED system is a deliberate
+  #     state (a cardinal sin was detected); restarting it would clear
+  #     the halt and lose the event. The monitor being ALIVE is what
+  #     liveness measures. No manna. No L3. No probe cycle.
+
   get "/kodesh/health" do
     health = build_health_check()
     status = if health.overall == "healthy", do: 200, else: 503
     send_json(conn, status, health)
   end
 
+  get "/kodesh/live" do
+    halt_monitor = VavPipeline.HaltMonitor.alive?()
+
+    halt_state =
+      case VavPipeline.HaltMonitor.status() do
+        :running -> "running"
+        {:halted, event} -> "halted: #{event.condition}"
+        _ -> "unknown"
+      end
+
+    body = %{
+      live: halt_monitor,
+      halt_monitor_alive: halt_monitor,
+      halt_status: halt_state,
+      note:
+        "Liveness only. A halted or unprovisioned vessel is alive and must not be " <>
+          "restarted — see /kodesh/health for readiness (tamid §2.1 check 3: manna absent → WAIT)."
+    }
+
+    send_json(conn, if(halt_monitor, do: 200, else: 503), body)
+  end
+
   get "/kodesh/status" do
     status = build_system_status()
     send_json(conn, 200, status)
@@ -239,7 +286,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do
   match _ do
     send_error(conn, 404, "NOT_FOUND", %{
       issue: "Unknown endpoint: #{conn.request_path}",
-      correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health",
+      correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health, /kodesh/live",
       resubmit: false
     })
   end