manna: the provision falls at the boundary — 00:00Z — and readiness is carried across it

The watchdog's first night (2026-10-21): 503 at 00:00:01Z with
manna_present and esh_tamid failing, recovered 00:05Z. Every night since
the CronJob was applied the site has been dark one to five minutes:
the provision is stale from 00:00:00, the job ran at 00:01, and the
readiness probe (10 s period, default threshold 3) fell in the gap.

Not changed: the rule. A provision is for its day; Friday's double is
the Shabbat's; nothing else crosses the boundary (Shemot 16:19–26,
manna_shabbat_test.exs). Changed: the timing.
- manna-cronjob.yaml: schedule "0 0 * * 0-5" — the manna falls at the
  boundary, not a minute after it.
- deployment.yaml: readinessProbe.failureThreshold 6 — 60 s of failures
  before the Service stops routing. The job writes within seconds of
  00:00; the pod is carried across. A real outage still shows within a
  minute; the watchdog's own threshold is ten.
- KOHEN.md: the night's finding and the new timing.

Both manifests validate (kubectl apply --dry-run=client). Proof: the
watchdog log on the next boundary should show no FAIL at 00:00.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Fable 5.1 committed 2026-10-21 17:53:45 +00:00
1 parent 489cf11416
commit 50b08cba4b
3 files changed
+22 -6

No files matched your search

+8 -3
View File
@@ -753,14 +753,19 @@ run on Saturday.
### The manna, each morning — *Dvar yom b'yomo*
Since 2026-10-15 the day's provision is written by the system's own hand:
CronJob `manna` (`deploy/k8s/manna-cronjob.yaml`) runs at 00:01Z Sunday
CronJob `manna` (`deploy/k8s/manna-cronjob.yaml`) runs at 00:00Z Sunday
to Friday, mounts the manna PVC and writes `today.json` with the amounts
from `shorashim-config` (`MANNA_COMPUTE_UNITS` …). Before it, every night
at 00:00Z the old provision went stale and the readiness probe failed
until someone woke (13.5 hours on 2026-10-15). The kohen's acts now:
until someone woke (13.5 hours on 2026-10-15). With the job at 00:01Z the
site was still 503 for one to five minutes every night (the watchdog saw
it, 2026-10-21): the provision is stale from 00:00:00 and readiness fell
before the job wrote. Now the job runs at 00:00Z and readiness waits 60 s
(`failureThreshold: 6`) before the Service stops routing — the manna
falls at the boundary, and the pod is carried across it. The kohen's acts now:
- set the amounts in the ConfigMap (`kubectl apply -f deploy/k8s/configmap.yaml`);
- read `kubectl -n shorashim get jobs` after 00:01Z — a failed job is the
- read `kubectl -n shorashim get jobs` after 00:00Z — a failed job is the
Day-4 alarm; write by hand with `kubectl cp` (Day 4) if it failed;
- run it at once when needed: `kubectl -n shorashim create job --from=cronjob/manna manna-now`.
+7
View File
@@ -266,6 +266,13 @@ spec:
port: 4000
initialDelaySeconds: 10
periodSeconds: 10
# 60 s of failures before the Service stops routing here (the
# default is 30 s). The manna goes stale at 00:00:00 and the
# CronJob writes the new day's within seconds; the cushion
# carries the pod across that boundary without a 503. A real
# outage still shows within a minute, and the watchdog's own
# threshold is ten.
failureThreshold: 6
# === SHORESH ENGINE ===
# Not a container. It runs as an Erlang Port inside
+7 -3
View File
@@ -3,7 +3,7 @@
# Until 2026-10-15 the day's provision was written by the kohen with
# `kubectl cp`, and every night at 00:00Z the old one went stale and the
# readiness probe failed until someone woke (13.5 hours on 2026-10-15).
# This CronJob writes today.json onto the manna PVC at 00:01Z, Sunday to
# This CronJob writes today.json onto the manna PVC at 00:00Z, Sunday to
# Friday — a double portion on Friday that is valid through Shabbat, and
# nothing on Shabbat (Shemot 16:22–26: "on the sixth day they gathered
# double … on the seventh day there shall be none"). The kohen still sets
@@ -23,10 +23,14 @@ metadata:
app.kubernetes.io/name: shorashim
app.kubernetes.io/component: manna
spec:
schedule: "1 0 * * 0-5" # 00:01Z, Sunday–Friday; none on Shabbat
schedule: "0 0 * * 0-5" # 00:00Z, Sunday–Friday; none on Shabbat. At the boundary, not after it:
# the provision goes stale at 00:00:00 and readiness has a 60 s cushion
# (deployment.yaml failureThreshold 6) — a job that writes within it never
# flips the pod. At 00:01 the site was 503 for 1–5 min every night
# (watchdog, 2026-10-21).
timeZone: "Etc/UTC"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 3600 # a missed 00:01 still runs within the hour
startingDeadlineSeconds: 3600 # a missed 00:00 still runs within the hour
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate: