probes: liveness asks "is it dead?", readiness asks "may it serve?" (kohen Day 3, restart 1)

TORAH ANCHOR — Shemot 16:4, and tamid §2.1 check 3:
  Manna provision read … If absent: WAIT. Alert senior kohanim.
WAIT. Not die. Provision is not the health of the vessel. Killing the
process does not produce manna; it re-runs init, re-spawns the Port,
and waits again — three minutes later.

WHAT THE KOHEN FOUND (BUILDER_HANDOFF_probes.md, 2026-09-07 20:35Z):

A fresh pod after recreate. emptyDir manna empty — by design
(KOHEN.md:406). manna_present honestly read "fail" — the Day-6 fix,
correct per tamid §1.1. overall → "degraded" → /kodesh/health 503.
BOTH probes pointed at /kodesh/health. Liveness: period 60s, default
failureThreshold 3 → after three minutes the kubelet SIGTERMed a
process whose log showed the Port alive, zero errors, l3_writable
pass, esh_tamid pass. It was alive and serving. It was killed for
lacking provision, and would be again every three minutes until the
kohen wrote today.json — i.e. on EVERY pod creation.

Introduced by a correct fix (f1ab7c8 made health honest); the missing
piece was the distinction between "not ready to serve" and "dead".

WHAT CHANGED:

bigdei_kodesh/router.ex
  GET /kodesh/live — 200 iff the BEAM answers HTTP and the HaltMonitor
  process is registered. Looks at NOTHING that is a wait state: no
  manna, no L3, no cloud, no probe cycle. And deliberately NOT "not
  halted": a HALTED system is a chosen state (a cardinal sin was
  detected); restarting it would clear the halt and lose the event.
  Liveness measures that the monitor is ALIVE, not that it is quiet.
  /kodesh/health is unchanged — it is readiness, and a vessel without
  provision correctly does not receive offerings (that is shema; the
  Service stops routing to it).

deploy/k8s/deployment.yaml
  livenessProbe → /kodesh/live, failureThreshold 3 made explicit.
  readinessProbe stays on /kodesh/health.

test/liveness_test.exs — 3 tests
  live 200 with NO manna while health is 503; live 200 while HALTED
  (and pe'ah answers 503 HALT_*, not a served result); live touches
  neither the ledger nor L3 nor the cycle.

KOHEN.md §Day 3c — a fresh pod reads 2/3 Ready, 0 restarts until Day 4
writes manna, and that is correct; restarts on a pod whose log shows
the Port alive and zero errors mean liveness is on the wrong endpoint.

ALSO — a leak of my own from f1ab7c8:
  The MannaLedger is a global GenServer keyed by date. The Day-6
  "provision exhausted" test left it at allocated 1 / consumed 1 for
  today; when step31 Test 4 later wrote a fresh today.json the ledger
  answered :already_loaded, consume failed, and the cycle waited —
  a 1-in-5-seeds flake. Fixed where it leaked (day6 + liveness tests
  reset the ledger on exit) and where it landed (step31 Test 4 gathers
  from a clean ledger, with the reason written down).

VERIFIED — the kohen's procedure, locally, DB deliberately dead:
  fresh, no manna     live 200   health 503 critical (manna + l3)
  manna written       live 200   health 503 degraded — l3_writable ONLY;
                                 manna_present flipped to pass
  manna removed       live 200   health 503
  exit/SIGTERM events in the container log: 0
  mix test — 440 tests, 0 failures, 36 excluded, across seeds 0 1 2 7 42
  kubectl apply --dry-run=client: deployment valid; liveness
    /kodesh/live (threshold 3), readiness /kodesh/health
  Doorpost.verify_identity() :ok — no @spec_files entry touched
  l1/integrity.sha256 unchanged: 99bbd07a…4ea377

RAISED, NOT FIXED: manna is an emptyDir, so every pod creation needs
the kohen to write it again. manna-volume.yaml's PVC exists for
exactly this and is "not applied at Day 3" by the runbook's own note.
That is an operations decision, not a probe one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KOHEN.md

@@ -235,6 +235,16 @@ Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `e
 
 #### 3c. Watch the pod come up
 
+**Two probes, two questions.** Liveness is `/kodesh/live` — "must this
+process be restarted?" — and looks only at whether the BEAM answers
+and the HaltMonitor is registered. Readiness is `/kodesh/health` —
+"may this vessel receive offerings?" — and fails on no manna, no L3,
+or a cloud in transition. So a fresh pod with no `today.json` reads
+**2/3 Ready, 0 restarts**, and stays that way until you write manna at
+Day 4. That is correct: tamid §2.1 check 3, manna absent → WAIT. If
+you see restarts on a pod whose log shows the Port alive and zero
+errors, the liveness probe is pointed at the wrong endpoint — builder.
+
 ```bash
 kubectl get pods -n shorashim -w
 ```

deploy/k8s/deployment.yaml

@@ -217,12 +217,25 @@ spec:
             limits:
               memory: "1Gi"
               cpu: "1000m"
+          # LIVENESS — "must this process be restarted?" — /kodesh/live.
+          # 200 iff the BEAM answers and the HaltMonitor is registered.
+          # Does NOT look at manna, L3, or the cloud. Kohen Day 3
+          # (restart 1): with liveness on /kodesh/health, a fresh pod
+          # whose emptyDir manna was (by design) empty read
+          # manna_present: fail → degraded → 503 ×3 → SIGTERM, and the
+          # kubelet killed a process that was alive and serving, every
+          # three minutes, until manna landed. tamid §2.1 check 3: manna
+          # absent → WAIT. Not die.
           livenessProbe:
             httpGet:
-              path: /kodesh/health
+              path: /kodesh/live
               port: 4000
             initialDelaySeconds: 30
             periodSeconds: 60
+            failureThreshold: 3
+          # READINESS — "may this vessel receive offerings?" —
+          # /kodesh/health. No manna, no L3, cloud in transition → 503
+          # and the Service stops routing here. That is shema. Unchanged.
           readinessProbe:
             httpGet:
               path: /kodesh/health

vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex

@@ -161,12 +161,59 @@ defmodule VavPipeline.BigdeiKodesh.Router do
 
   # === KODESH — Operator Interface ===
 
+  # === KODESH — Readiness vs. liveness ===
+  #
+  # Two questions, two endpoints. The kohen's Day-3 (restart 1) finding:
+  # both probes pointed at /kodesh/health. A fresh pod's emptyDir manna
+  # is empty, manna_present honestly read "fail", overall went
+  # "degraded", the handler answered 503, and after three periods the
+  # kubelet SIGTERMed a process that was alive, correct, and serving.
+  # Every pod creation entered a 3-minute kill loop until manna landed.
+  #
+  # tamid §2.1 check 3: manna absent → WAIT. Not die. Provision is not
+  # the health of the vessel. Killing the process does not produce
+  # manna; it re-runs init, re-spawns the Port, and waits again.
+  #
+  #   /kodesh/health — READINESS. "May this vessel receive offerings?"
+  #     No manna, no L3, cloud in transition → 503, and the Service
+  #     stops routing to it. That is shema: a vessel without provision
+  #     does not serve. Correct, and unchanged.
+  #
+  #   /kodesh/live   — LIVENESS. "Must this process be restarted?"
+  #     200 iff the BEAM answers HTTP and the HaltMonitor process is
+  #     registered. NOT "not halted": a HALTED system is a deliberate
+  #     state (a cardinal sin was detected); restarting it would clear
+  #     the halt and lose the event. The monitor being ALIVE is what
+  #     liveness measures. No manna. No L3. No probe cycle.
+
   get "/kodesh/health" do
     health = build_health_check()
     status = if health.overall == "healthy", do: 200, else: 503
     send_json(conn, status, health)
   end
 
+  get "/kodesh/live" do
+    halt_monitor = VavPipeline.HaltMonitor.alive?()
+
+    halt_state =
+      case VavPipeline.HaltMonitor.status() do
+        :running -> "running"
+        {:halted, event} -> "halted: #{event.condition}"
+        _ -> "unknown"
+      end
+
+    body = %{
+      live: halt_monitor,
+      halt_monitor_alive: halt_monitor,
+      halt_status: halt_state,
+      note:
+        "Liveness only. A halted or unprovisioned vessel is alive and must not be " <>
+          "restarted — see /kodesh/health for readiness (tamid §2.1 check 3: manna absent → WAIT)."
+    }
+
+    send_json(conn, if(halt_monitor, do: 200, else: 503), body)
+  end
+
   get "/kodesh/status" do
     status = build_system_status()
     send_json(conn, 200, status)
@@ -239,7 +286,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do
   match _ do
     send_error(conn, 404, "NOT_FOUND", %{
       issue: "Unknown endpoint: #{conn.request_path}",
-      correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health",
+      correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health, /kodesh/live",
       resubmit: false
     })
   end

vav_pipeline/test/day6_serving_test.exs

@@ -110,8 +110,13 @@ defmodule VavPipeline.Day6ServingTest do
     ctx
   end
 
+  # The ledger is a global GenServer. Every test that loads or consumes
+  # from it must leave it as it found it — an exhausted ledger leaked
+  # into an older test (step31, Test 4) reads :already_loaded, refuses
+  # the fresh provision that test writes, and the cycle waits.
   defp write_provision(compute_units) do
     MannaLedger.reset()
+    on_exit(fn -> MannaLedger.reset() end)
     today = Date.utc_today() |> Date.to_iso8601()
     expires = DateTime.utc_now() |> DateTime.add(6 * 3600) |> DateTime.to_iso8601()
 

vav_pipeline/test/liveness_test.exs

@@ -0,0 +1,71 @@
+defmodule VavPipeline.LivenessTest do
+  @moduledoc """
+  Kohen Day 3, restart 1 (2026-09-07): both probes pointed at
+  /kodesh/health; a fresh pod with no manna read "degraded" → 503 → the
+  kubelet killed a live, serving process every three minutes until the
+  kohen wrote today.json.
+
+  tamid §2.1 check 3: manna absent → WAIT. Not die.
+
+  /kodesh/live answers one question — must this process be restarted —
+  and looks at nothing that is a wait state.
+  """
+
+  use ExUnit.Case, async: false
+  use Plug.Test
+
+  alias VavPipeline.BigdeiKodesh.Router
+  alias VavPipeline.HaltMonitor
+
+  @opts Router.init([])
+
+  setup do
+    # Make sure no provision exists — the exact state of a fresh pod.
+    File.rm(VavPipeline.Manna.provision_file())
+    :ok
+  end
+
+  test "live is 200 with NO manna, while health is 503 (wait, not die)" do
+    health = conn(:get, "/kodesh/health") |> Router.call(@opts)
+    assert health.status == 503
+    assert Jason.decode!(health.resp_body)["checks"]["manna_present"] == "fail"
+
+    live = conn(:get, "/kodesh/live") |> Router.call(@opts)
+    assert live.status == 200
+    body = Jason.decode!(live.resp_body)
+    assert body["live"] == true
+    assert body["halt_monitor_alive"] == true
+    assert body["halt_status"] == "running"
+  end
+
+  test "live is 200 while HALTED — a halt is a state, not a death" do
+    HaltMonitor.halt!(:avodah_zarah, "liveness test", "test")
+
+    try do
+      assert HaltMonitor.halted?()
+
+      live = conn(:get, "/kodesh/live") |> Router.call(@opts)
+      assert live.status == 200
+      assert Jason.decode!(live.resp_body)["halt_status"] =~ "halted: avodah_zarah"
+
+      # And the public ring says so, rather than serving.
+      peah =
+        conn(:post, "/peah/ask", Jason.encode!(%{question: "בראשית ברא אלהים"}))
+        |> put_req_header("content-type", "application/json")
+        |> Router.call(@opts)
+
+      assert peah.status == 503
+      assert Jason.decode!(peah.resp_body)["error"]["code"] =~ "HALT_"
+    after
+      HaltMonitor.resolve("liveness test cleanup")
+    end
+  end
+
+  test "live touches neither manna nor L3 nor the cycle" do
+    VavPipeline.MannaLedger.reset()
+    on_exit(fn -> VavPipeline.MannaLedger.reset() end)
+    before = VavPipeline.MannaLedger.ledger()
+    conn(:get, "/kodesh/live") |> Router.call(@opts)
+    assert VavPipeline.MannaLedger.ledger() == before
+  end
+end

vav_pipeline/test/step31_shema_test.exs

@@ -316,6 +316,13 @@ defmodule VavPipeline.Step31.ShemaTest do
     end
 
     test "provision file present → system processes" do
+      # The cycle now CONSUMES from the MannaLedger (kohen Day 6). The
+      # ledger is a global GenServer keyed by date: a provision another
+      # test loaded and exhausted today would read :already_loaded and
+      # refuse the fresh file written below. Gather from a clean ledger.
+      VavPipeline.MannaLedger.reset()
+      on_exit(fn -> VavPipeline.MannaLedger.reset() end)
+
       # Write a valid provision file
       today = Date.utc_today() |> Date.to_iso8601()