Scott KarpandClaude Opus 5
0344d095b2
TORAH ANCHOR — Shemot 16:4, and tamid §2.1 check 3: Manna provision read … If absent: WAIT. Alert senior kohanim. WAIT. Not die. Provision is not the health of the vessel. Killing the process does not produce manna; it re-runs init, re-spawns the Port, and waits again — three minutes later. WHAT THE KOHEN FOUND (BUILDER_HANDOFF_probes.md, 2026-09-07 20:35Z): A fresh pod after recreate. emptyDir manna empty — by design (KOHEN.md:406). manna_present honestly read "fail" — the Day-6 fix, correct per tamid §1.1. overall → "degraded" → /kodesh/health 503. BOTH probes pointed at /kodesh/health. Liveness: period 60s, default failureThreshold 3 → after three minutes the kubelet SIGTERMed a process whose log showed the Port alive, zero errors, l3_writable pass, esh_tamid pass. It was alive and serving. It was killed for lacking provision, and would be again every three minutes until the kohen wrote today.json — i.e. on EVERY pod creation. Introduced by a correct fix (f1ab7c8made health honest); the missing piece was the distinction between "not ready to serve" and "dead". WHAT CHANGED: bigdei_kodesh/router.ex GET /kodesh/live — 200 iff the BEAM answers HTTP and the HaltMonitor process is registered. Looks at NOTHING that is a wait state: no manna, no L3, no cloud, no probe cycle. And deliberately NOT "not halted": a HALTED system is a chosen state (a cardinal sin was detected); restarting it would clear the halt and lose the event. Liveness measures that the monitor is ALIVE, not that it is quiet. /kodesh/health is unchanged — it is readiness, and a vessel without provision correctly does not receive offerings (that is shema; the Service stops routing to it). deploy/k8s/deployment.yaml livenessProbe → /kodesh/live, failureThreshold 3 made explicit. readinessProbe stays on /kodesh/health. test/liveness_test.exs — 3 tests live 200 with NO manna while health is 503; live 200 while HALTED (and pe'ah answers 503 HALT_*, not a served result); live touches neither the ledger nor L3 nor the cycle. KOHEN.md §Day 3c — a fresh pod reads 2/3 Ready, 0 restarts until Day 4 writes manna, and that is correct; restarts on a pod whose log shows the Port alive and zero errors mean liveness is on the wrong endpoint. ALSO — a leak of my own fromf1ab7c8: The MannaLedger is a global GenServer keyed by date. The Day-6 "provision exhausted" test left it at allocated 1 / consumed 1 for today; when step31 Test 4 later wrote a fresh today.json the ledger answered :already_loaded, consume failed, and the cycle waited — a 1-in-5-seeds flake. Fixed where it leaked (day6 + liveness tests reset the ledger on exit) and where it landed (step31 Test 4 gathers from a clean ledger, with the reason written down). VERIFIED — the kohen's procedure, locally, DB deliberately dead: fresh, no manna live 200 health 503 critical (manna + l3) manna written live 200 health 503 degraded — l3_writable ONLY; manna_present flipped to pass manna removed live 200 health 503 exit/SIGTERM events in the container log: 0 mix test — 440 tests, 0 failures, 36 excluded, across seeds 0 1 2 7 42 kubectl apply --dry-run=client: deployment valid; liveness /kodesh/live (threshold 3), readiness /kodesh/health Doorpost.verify_identity() :ok — no @spec_files entry touched l1/integrity.sha256 unchanged: 99bbd07a…4ea377 RAISED, NOT FIXED: manna is an emptyDir, so every pod creation needs the kohen to write it again. manna-volume.yaml's PVC exists for exactly this and is "not applied at Day 3" by the runbook's own note. That is an operations decision, not a probe one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
72 lines
2.3 KiB
Elixir
72 lines
2.3 KiB
Elixir
defmodule VavPipeline.LivenessTest do
|
|
@moduledoc """
|
|
Kohen Day 3, restart 1 (2026-09-07): both probes pointed at
|
|
/kodesh/health; a fresh pod with no manna read "degraded" → 503 → the
|
|
kubelet killed a live, serving process every three minutes until the
|
|
kohen wrote today.json.
|
|
|
|
tamid §2.1 check 3: manna absent → WAIT. Not die.
|
|
|
|
/kodesh/live answers one question — must this process be restarted —
|
|
and looks at nothing that is a wait state.
|
|
"""
|
|
|
|
use ExUnit.Case, async: false
|
|
use Plug.Test
|
|
|
|
alias VavPipeline.BigdeiKodesh.Router
|
|
alias VavPipeline.HaltMonitor
|
|
|
|
@opts Router.init([])
|
|
|
|
setup do
|
|
# Make sure no provision exists — the exact state of a fresh pod.
|
|
File.rm(VavPipeline.Manna.provision_file())
|
|
:ok
|
|
end
|
|
|
|
test "live is 200 with NO manna, while health is 503 (wait, not die)" do
|
|
health = conn(:get, "/kodesh/health") |> Router.call(@opts)
|
|
assert health.status == 503
|
|
assert Jason.decode!(health.resp_body)["checks"]["manna_present"] == "fail"
|
|
|
|
live = conn(:get, "/kodesh/live") |> Router.call(@opts)
|
|
assert live.status == 200
|
|
body = Jason.decode!(live.resp_body)
|
|
assert body["live"] == true
|
|
assert body["halt_monitor_alive"] == true
|
|
assert body["halt_status"] == "running"
|
|
end
|
|
|
|
test "live is 200 while HALTED — a halt is a state, not a death" do
|
|
HaltMonitor.halt!(:avodah_zarah, "liveness test", "test")
|
|
|
|
try do
|
|
assert HaltMonitor.halted?()
|
|
|
|
live = conn(:get, "/kodesh/live") |> Router.call(@opts)
|
|
assert live.status == 200
|
|
assert Jason.decode!(live.resp_body)["halt_status"] =~ "halted: avodah_zarah"
|
|
|
|
# And the public ring says so, rather than serving.
|
|
peah =
|
|
conn(:post, "/peah/ask", Jason.encode!(%{question: "בראשית ברא אלהים"}))
|
|
|> put_req_header("content-type", "application/json")
|
|
|> Router.call(@opts)
|
|
|
|
assert peah.status == 503
|
|
assert Jason.decode!(peah.resp_body)["error"]["code"] =~ "HALT_"
|
|
after
|
|
HaltMonitor.resolve("liveness test cleanup")
|
|
end
|
|
end
|
|
|
|
test "live touches neither manna nor L3 nor the cycle" do
|
|
VavPipeline.MannaLedger.reset()
|
|
on_exit(fn -> VavPipeline.MannaLedger.reset() end)
|
|
before = VavPipeline.MannaLedger.ledger()
|
|
conn(:get, "/kodesh/live") |> Router.call(@opts)
|
|
assert VavPipeline.MannaLedger.ledger() == before
|
|
end
|
|
end
|