From 50b08cba4bd9143f6010271ad4fb042448a63c0c Mon Sep 17 00:00:00 2001 From: Scott Karp Date: Wed, 21 Oct 2026 17:53:45 +0000 Subject: [PATCH] =?UTF-8?q?manna:=20the=20provision=20falls=20at=20the=20b?= =?UTF-8?q?oundary=20=E2=80=94=2000:00Z=20=E2=80=94=20and=20readiness=20is?= =?UTF-8?q?=20carried=20across=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The watchdog's first night (2026-10-21): 503 at 00:00:01Z with manna_present and esh_tamid failing, recovered 00:05Z. Every night since the CronJob was applied the site has been dark one to five minutes: the provision is stale from 00:00:00, the job ran at 00:01, and the readiness probe (10 s period, default threshold 3) fell in the gap. Not changed: the rule. A provision is for its day; Friday's double is the Shabbat's; nothing else crosses the boundary (Shemot 16:19–26, manna_shabbat_test.exs). Changed: the timing. - manna-cronjob.yaml: schedule "0 0 * * 0-5" — the manna falls at the boundary, not a minute after it. - deployment.yaml: readinessProbe.failureThreshold 6 — 60 s of failures before the Service stops routing. The job writes within seconds of 00:00; the pod is carried across. A real outage still shows within a minute; the watchdog's own threshold is ten. - KOHEN.md: the night's finding and the new timing. Both manifests validate (kubectl apply --dry-run=client). Proof: the watchdog log on the next boundary should show no FAIL at 00:00. Co-Authored-By: Claude Fable 5.1 --- KOHEN.md | 11 ++++++++--- deploy/k8s/deployment.yaml | 7 +++++++ deploy/k8s/manna-cronjob.yaml | 10 +++++++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/KOHEN.md b/KOHEN.md index 57d6fbd6..0f27853d 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -753,14 +753,19 @@ run on Saturday. ### The manna, each morning — *Dvar yom b'yomo* Since 2026-10-15 the day's provision is written by the system's own hand: -CronJob `manna` (`deploy/k8s/manna-cronjob.yaml`) runs at 00:01Z Sunday +CronJob `manna` (`deploy/k8s/manna-cronjob.yaml`) runs at 00:00Z Sunday to Friday, mounts the manna PVC and writes `today.json` with the amounts from `shorashim-config` (`MANNA_COMPUTE_UNITS` …). Before it, every night at 00:00Z the old provision went stale and the readiness probe failed -until someone woke (13.5 hours on 2026-10-15). The kohen's acts now: +until someone woke (13.5 hours on 2026-10-15). With the job at 00:01Z the +site was still 503 for one to five minutes every night (the watchdog saw +it, 2026-10-21): the provision is stale from 00:00:00 and readiness fell +before the job wrote. Now the job runs at 00:00Z and readiness waits 60 s +(`failureThreshold: 6`) before the Service stops routing — the manna +falls at the boundary, and the pod is carried across it. The kohen's acts now: - set the amounts in the ConfigMap (`kubectl apply -f deploy/k8s/configmap.yaml`); -- read `kubectl -n shorashim get jobs` after 00:01Z — a failed job is the +- read `kubectl -n shorashim get jobs` after 00:00Z — a failed job is the Day-4 alarm; write by hand with `kubectl cp` (Day 4) if it failed; - run it at once when needed: `kubectl -n shorashim create job --from=cronjob/manna manna-now`. diff --git a/deploy/k8s/deployment.yaml b/deploy/k8s/deployment.yaml index 1216bf80..043400e2 100644 --- a/deploy/k8s/deployment.yaml +++ b/deploy/k8s/deployment.yaml @@ -266,6 +266,13 @@ spec: port: 4000 initialDelaySeconds: 10 periodSeconds: 10 + # 60 s of failures before the Service stops routing here (the + # default is 30 s). The manna goes stale at 00:00:00 and the + # CronJob writes the new day's within seconds; the cushion + # carries the pod across that boundary without a 503. A real + # outage still shows within a minute, and the watchdog's own + # threshold is ten. + failureThreshold: 6 # === SHORESH ENGINE === # Not a container. It runs as an Erlang Port inside diff --git a/deploy/k8s/manna-cronjob.yaml b/deploy/k8s/manna-cronjob.yaml index 559a219b..bb38035d 100644 --- a/deploy/k8s/manna-cronjob.yaml +++ b/deploy/k8s/manna-cronjob.yaml @@ -3,7 +3,7 @@ # Until 2026-10-15 the day's provision was written by the kohen with # `kubectl cp`, and every night at 00:00Z the old one went stale and the # readiness probe failed until someone woke (13.5 hours on 2026-10-15). -# This CronJob writes today.json onto the manna PVC at 00:01Z, Sunday to +# This CronJob writes today.json onto the manna PVC at 00:00Z, Sunday to # Friday — a double portion on Friday that is valid through Shabbat, and # nothing on Shabbat (Shemot 16:22–26: "on the sixth day they gathered # double … on the seventh day there shall be none"). The kohen still sets @@ -23,10 +23,14 @@ metadata: app.kubernetes.io/name: shorashim app.kubernetes.io/component: manna spec: - schedule: "1 0 * * 0-5" # 00:01Z, Sunday–Friday; none on Shabbat + schedule: "0 0 * * 0-5" # 00:00Z, Sunday–Friday; none on Shabbat. At the boundary, not after it: + # the provision goes stale at 00:00:00 and readiness has a 60 s cushion + # (deployment.yaml failureThreshold 6) — a job that writes within it never + # flips the pod. At 00:01 the site was 503 for 1–5 min every night + # (watchdog, 2026-10-21). timeZone: "Etc/UTC" concurrencyPolicy: Forbid - startingDeadlineSeconds: 3600 # a missed 00:01 still runs within the hour + startingDeadlineSeconds: 3600 # a missed 00:00 still runs within the hour successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: