From 0344d095b29b783613fc5823fa91ac69abf4ee89 Mon Sep 17 00:00:00 2001 From: Scott Karp Date: Mon, 7 Sep 2026 17:04:43 -0400 Subject: [PATCH] probes: liveness asks "is it dead?", readiness asks "may it serve?" (kohen Day 3, restart 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TORAH ANCHOR — Shemot 16:4, and tamid §2.1 check 3: Manna provision read … If absent: WAIT. Alert senior kohanim. WAIT. Not die. Provision is not the health of the vessel. Killing the process does not produce manna; it re-runs init, re-spawns the Port, and waits again — three minutes later. WHAT THE KOHEN FOUND (BUILDER_HANDOFF_probes.md, 2026-09-07 20:35Z): A fresh pod after recreate. emptyDir manna empty — by design (KOHEN.md:406). manna_present honestly read "fail" — the Day-6 fix, correct per tamid §1.1. overall → "degraded" → /kodesh/health 503. BOTH probes pointed at /kodesh/health. Liveness: period 60s, default failureThreshold 3 → after three minutes the kubelet SIGTERMed a process whose log showed the Port alive, zero errors, l3_writable pass, esh_tamid pass. It was alive and serving. It was killed for lacking provision, and would be again every three minutes until the kohen wrote today.json — i.e. on EVERY pod creation. Introduced by a correct fix (f1ab7c8 made health honest); the missing piece was the distinction between "not ready to serve" and "dead". WHAT CHANGED: bigdei_kodesh/router.ex GET /kodesh/live — 200 iff the BEAM answers HTTP and the HaltMonitor process is registered. Looks at NOTHING that is a wait state: no manna, no L3, no cloud, no probe cycle. And deliberately NOT "not halted": a HALTED system is a chosen state (a cardinal sin was detected); restarting it would clear the halt and lose the event. Liveness measures that the monitor is ALIVE, not that it is quiet. /kodesh/health is unchanged — it is readiness, and a vessel without provision correctly does not receive offerings (that is shema; the Service stops routing to it). deploy/k8s/deployment.yaml livenessProbe → /kodesh/live, failureThreshold 3 made explicit. readinessProbe stays on /kodesh/health. test/liveness_test.exs — 3 tests live 200 with NO manna while health is 503; live 200 while HALTED (and pe'ah answers 503 HALT_*, not a served result); live touches neither the ledger nor L3 nor the cycle. KOHEN.md §Day 3c — a fresh pod reads 2/3 Ready, 0 restarts until Day 4 writes manna, and that is correct; restarts on a pod whose log shows the Port alive and zero errors mean liveness is on the wrong endpoint. ALSO — a leak of my own from f1ab7c8: The MannaLedger is a global GenServer keyed by date. The Day-6 "provision exhausted" test left it at allocated 1 / consumed 1 for today; when step31 Test 4 later wrote a fresh today.json the ledger answered :already_loaded, consume failed, and the cycle waited — a 1-in-5-seeds flake. Fixed where it leaked (day6 + liveness tests reset the ledger on exit) and where it landed (step31 Test 4 gathers from a clean ledger, with the reason written down). VERIFIED — the kohen's procedure, locally, DB deliberately dead: fresh, no manna live 200 health 503 critical (manna + l3) manna written live 200 health 503 degraded — l3_writable ONLY; manna_present flipped to pass manna removed live 200 health 503 exit/SIGTERM events in the container log: 0 mix test — 440 tests, 0 failures, 36 excluded, across seeds 0 1 2 7 42 kubectl apply --dry-run=client: deployment valid; liveness /kodesh/live (threshold 3), readiness /kodesh/health Doorpost.verify_identity() :ok — no @spec_files entry touched l1/integrity.sha256 unchanged: 99bbd07a…4ea377 RAISED, NOT FIXED: manna is an emptyDir, so every pod creation needs the kohen to write it again. manna-volume.yaml's PVC exists for exactly this and is "not applied at Day 3" by the runbook's own note. That is an operations decision, not a probe one. Co-Authored-By: Claude Opus 5 --- KOHEN.md | 10 +++ deploy/k8s/deployment.yaml | 15 +++- .../lib/vav_pipeline/bigdei_kodesh/router.ex | 49 ++++++++++++- vav_pipeline/test/day6_serving_test.exs | 5 ++ vav_pipeline/test/liveness_test.exs | 71 +++++++++++++++++++ vav_pipeline/test/step31_shema_test.exs | 7 ++ 6 files changed, 155 insertions(+), 2 deletions(-) create mode 100644 vav_pipeline/test/liveness_test.exs diff --git a/KOHEN.md b/KOHEN.md index 711672c0..a89f0ca1 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -235,6 +235,16 @@ Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `e #### 3c. Watch the pod come up +**Two probes, two questions.** Liveness is `/kodesh/live` — "must this +process be restarted?" — and looks only at whether the BEAM answers +and the HaltMonitor is registered. Readiness is `/kodesh/health` — +"may this vessel receive offerings?" — and fails on no manna, no L3, +or a cloud in transition. So a fresh pod with no `today.json` reads +**2/3 Ready, 0 restarts**, and stays that way until you write manna at +Day 4. That is correct: tamid §2.1 check 3, manna absent → WAIT. If +you see restarts on a pod whose log shows the Port alive and zero +errors, the liveness probe is pointed at the wrong endpoint — builder. + ```bash kubectl get pods -n shorashim -w ``` diff --git a/deploy/k8s/deployment.yaml b/deploy/k8s/deployment.yaml index 99ec9ba4..54ffb65c 100644 --- a/deploy/k8s/deployment.yaml +++ b/deploy/k8s/deployment.yaml @@ -217,12 +217,25 @@ spec: limits: memory: "1Gi" cpu: "1000m" + # LIVENESS — "must this process be restarted?" — /kodesh/live. + # 200 iff the BEAM answers and the HaltMonitor is registered. + # Does NOT look at manna, L3, or the cloud. Kohen Day 3 + # (restart 1): with liveness on /kodesh/health, a fresh pod + # whose emptyDir manna was (by design) empty read + # manna_present: fail → degraded → 503 ×3 → SIGTERM, and the + # kubelet killed a process that was alive and serving, every + # three minutes, until manna landed. tamid §2.1 check 3: manna + # absent → WAIT. Not die. livenessProbe: httpGet: - path: /kodesh/health + path: /kodesh/live port: 4000 initialDelaySeconds: 30 periodSeconds: 60 + failureThreshold: 3 + # READINESS — "may this vessel receive offerings?" — + # /kodesh/health. No manna, no L3, cloud in transition → 503 + # and the Service stops routing here. That is shema. Unchanged. readinessProbe: httpGet: path: /kodesh/health diff --git a/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex b/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex index 1e2b11e4..443504f4 100644 --- a/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex +++ b/vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex @@ -161,12 +161,59 @@ defmodule VavPipeline.BigdeiKodesh.Router do # === KODESH — Operator Interface === + # === KODESH — Readiness vs. liveness === + # + # Two questions, two endpoints. The kohen's Day-3 (restart 1) finding: + # both probes pointed at /kodesh/health. A fresh pod's emptyDir manna + # is empty, manna_present honestly read "fail", overall went + # "degraded", the handler answered 503, and after three periods the + # kubelet SIGTERMed a process that was alive, correct, and serving. + # Every pod creation entered a 3-minute kill loop until manna landed. + # + # tamid §2.1 check 3: manna absent → WAIT. Not die. Provision is not + # the health of the vessel. Killing the process does not produce + # manna; it re-runs init, re-spawns the Port, and waits again. + # + # /kodesh/health — READINESS. "May this vessel receive offerings?" + # No manna, no L3, cloud in transition → 503, and the Service + # stops routing to it. That is shema: a vessel without provision + # does not serve. Correct, and unchanged. + # + # /kodesh/live — LIVENESS. "Must this process be restarted?" + # 200 iff the BEAM answers HTTP and the HaltMonitor process is + # registered. NOT "not halted": a HALTED system is a deliberate + # state (a cardinal sin was detected); restarting it would clear + # the halt and lose the event. The monitor being ALIVE is what + # liveness measures. No manna. No L3. No probe cycle. + get "/kodesh/health" do health = build_health_check() status = if health.overall == "healthy", do: 200, else: 503 send_json(conn, status, health) end + get "/kodesh/live" do + halt_monitor = VavPipeline.HaltMonitor.alive?() + + halt_state = + case VavPipeline.HaltMonitor.status() do + :running -> "running" + {:halted, event} -> "halted: #{event.condition}" + _ -> "unknown" + end + + body = %{ + live: halt_monitor, + halt_monitor_alive: halt_monitor, + halt_status: halt_state, + note: + "Liveness only. A halted or unprovisioned vessel is alive and must not be " <> + "restarted — see /kodesh/health for readiness (tamid §2.1 check 3: manna absent → WAIT)." + } + + send_json(conn, if(halt_monitor, do: 200, else: 503), body) + end + get "/kodesh/status" do status = build_system_status() send_json(conn, 200, status) @@ -239,7 +286,7 @@ defmodule VavPipeline.BigdeiKodesh.Router do match _ do send_error(conn, 404, "NOT_FOUND", %{ issue: "Unknown endpoint: #{conn.request_path}", - correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health", + correction: "Available: /peah/ask, /courtyard/offering, /chatzer/sanhedrin, /kodesh/health, /kodesh/live", resubmit: false }) end diff --git a/vav_pipeline/test/day6_serving_test.exs b/vav_pipeline/test/day6_serving_test.exs index d76d6009..a08db72a 100644 --- a/vav_pipeline/test/day6_serving_test.exs +++ b/vav_pipeline/test/day6_serving_test.exs @@ -110,8 +110,13 @@ defmodule VavPipeline.Day6ServingTest do ctx end + # The ledger is a global GenServer. Every test that loads or consumes + # from it must leave it as it found it — an exhausted ledger leaked + # into an older test (step31, Test 4) reads :already_loaded, refuses + # the fresh provision that test writes, and the cycle waits. defp write_provision(compute_units) do MannaLedger.reset() + on_exit(fn -> MannaLedger.reset() end) today = Date.utc_today() |> Date.to_iso8601() expires = DateTime.utc_now() |> DateTime.add(6 * 3600) |> DateTime.to_iso8601() diff --git a/vav_pipeline/test/liveness_test.exs b/vav_pipeline/test/liveness_test.exs new file mode 100644 index 00000000..5e7672cc --- /dev/null +++ b/vav_pipeline/test/liveness_test.exs @@ -0,0 +1,71 @@ +defmodule VavPipeline.LivenessTest do + @moduledoc """ + Kohen Day 3, restart 1 (2026-09-07): both probes pointed at + /kodesh/health; a fresh pod with no manna read "degraded" → 503 → the + kubelet killed a live, serving process every three minutes until the + kohen wrote today.json. + + tamid §2.1 check 3: manna absent → WAIT. Not die. + + /kodesh/live answers one question — must this process be restarted — + and looks at nothing that is a wait state. + """ + + use ExUnit.Case, async: false + use Plug.Test + + alias VavPipeline.BigdeiKodesh.Router + alias VavPipeline.HaltMonitor + + @opts Router.init([]) + + setup do + # Make sure no provision exists — the exact state of a fresh pod. + File.rm(VavPipeline.Manna.provision_file()) + :ok + end + + test "live is 200 with NO manna, while health is 503 (wait, not die)" do + health = conn(:get, "/kodesh/health") |> Router.call(@opts) + assert health.status == 503 + assert Jason.decode!(health.resp_body)["checks"]["manna_present"] == "fail" + + live = conn(:get, "/kodesh/live") |> Router.call(@opts) + assert live.status == 200 + body = Jason.decode!(live.resp_body) + assert body["live"] == true + assert body["halt_monitor_alive"] == true + assert body["halt_status"] == "running" + end + + test "live is 200 while HALTED — a halt is a state, not a death" do + HaltMonitor.halt!(:avodah_zarah, "liveness test", "test") + + try do + assert HaltMonitor.halted?() + + live = conn(:get, "/kodesh/live") |> Router.call(@opts) + assert live.status == 200 + assert Jason.decode!(live.resp_body)["halt_status"] =~ "halted: avodah_zarah" + + # And the public ring says so, rather than serving. + peah = + conn(:post, "/peah/ask", Jason.encode!(%{question: "בראשית ברא אלהים"})) + |> put_req_header("content-type", "application/json") + |> Router.call(@opts) + + assert peah.status == 503 + assert Jason.decode!(peah.resp_body)["error"]["code"] =~ "HALT_" + after + HaltMonitor.resolve("liveness test cleanup") + end + end + + test "live touches neither manna nor L3 nor the cycle" do + VavPipeline.MannaLedger.reset() + on_exit(fn -> VavPipeline.MannaLedger.reset() end) + before = VavPipeline.MannaLedger.ledger() + conn(:get, "/kodesh/live") |> Router.call(@opts) + assert VavPipeline.MannaLedger.ledger() == before + end +end diff --git a/vav_pipeline/test/step31_shema_test.exs b/vav_pipeline/test/step31_shema_test.exs index cebad5b1..08ed806b 100644 --- a/vav_pipeline/test/step31_shema_test.exs +++ b/vav_pipeline/test/step31_shema_test.exs @@ -316,6 +316,13 @@ defmodule VavPipeline.Step31.ShemaTest do end test "provision file present → system processes" do + # The cycle now CONSUMES from the MannaLedger (kohen Day 6). The + # ledger is a global GenServer keyed by date: a provision another + # test loaded and exhausted today would read :already_loaded and + # refuse the fresh file written below. Gather from a clean ledger. + VavPipeline.MannaLedger.reset() + on_exit(fn -> VavPipeline.MannaLedger.reset() end) + # Write a valid provision file today = Date.utc_today() |> Date.to_iso8601()