Scott KarpandClaude Fable 5.1
50b08cba4b
The watchdog's first night (2026-10-21): 503 at 00:00:01Z with manna_present and esh_tamid failing, recovered 00:05Z. Every night since the CronJob was applied the site has been dark one to five minutes: the provision is stale from 00:00:00, the job ran at 00:01, and the readiness probe (10 s period, default threshold 3) fell in the gap. Not changed: the rule. A provision is for its day; Friday's double is the Shabbat's; nothing else crosses the boundary (Shemot 16:19–26, manna_shabbat_test.exs). Changed: the timing. - manna-cronjob.yaml: schedule "0 0 * * 0-5" — the manna falls at the boundary, not a minute after it. - deployment.yaml: readinessProbe.failureThreshold 6 — 60 s of failures before the Service stops routing. The job writes within seconds of 00:00; the pod is carried across. A real outage still shows within a minute; the watchdog's own threshold is ten. - KOHEN.md: the night's finding and the new timing. Both manifests validate (kubectl apply --dry-run=client). Proof: the watchdog log on the next boundary should show no FAIL at 00:00. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
101 lines
4.8 KiB
YAML
101 lines
4.8 KiB
YAML
# The manna — written each morning by the system's own hand (Shemot 16).
|
||
#
|
||
# Until 2026-10-15 the day's provision was written by the kohen with
|
||
# `kubectl cp`, and every night at 00:00Z the old one went stale and the
|
||
# readiness probe failed until someone woke (13.5 hours on 2026-10-15).
|
||
# This CronJob writes today.json onto the manna PVC at 00:00Z, Sunday to
|
||
# Friday — a double portion on Friday that is valid through Shabbat, and
|
||
# nothing on Shabbat (Shemot 16:22–26: "on the sixth day they gathered
|
||
# double … on the seventh day there shall be none"). The kohen still sets
|
||
# the amounts (shorashim-config, MANNA_* below) and may still write a
|
||
# provision by hand at any hour; the CronJob only ends the gap.
|
||
#
|
||
# It runs our own korban-processor image (python, pinned to a commit tag)
|
||
# with the PVC mounted, as the app's uid/gid, so the file is readable by
|
||
# the pod; nothing else, no cluster permissions. ReadWriteOnce is fine:
|
||
# the cluster has one node and the job runs beside the pod.
|
||
apiVersion: batch/v1
|
||
kind: CronJob
|
||
metadata:
|
||
name: manna
|
||
namespace: shorashim
|
||
labels:
|
||
app.kubernetes.io/name: shorashim
|
||
app.kubernetes.io/component: manna
|
||
spec:
|
||
schedule: "0 0 * * 0-5" # 00:00Z, Sunday–Friday; none on Shabbat. At the boundary, not after it:
|
||
# the provision goes stale at 00:00:00 and readiness has a 60 s cushion
|
||
# (deployment.yaml failureThreshold 6) — a job that writes within it never
|
||
# flips the pod. At 00:01 the site was 503 for 1–5 min every night
|
||
# (watchdog, 2026-10-21).
|
||
timeZone: "Etc/UTC"
|
||
concurrencyPolicy: Forbid
|
||
startingDeadlineSeconds: 3600 # a missed 00:00 still runs within the hour
|
||
successfulJobsHistoryLimit: 3
|
||
failedJobsHistoryLimit: 3
|
||
jobTemplate:
|
||
spec:
|
||
backoffLimit: 2
|
||
template:
|
||
metadata:
|
||
labels:
|
||
app.kubernetes.io/name: shorashim
|
||
app.kubernetes.io/component: manna
|
||
spec:
|
||
restartPolicy: Never
|
||
securityContext:
|
||
runAsUser: 1000
|
||
runAsGroup: 1000
|
||
fsGroup: 1000
|
||
imagePullSecrets:
|
||
- name: shorashim-registry
|
||
volumes:
|
||
- name: manna
|
||
persistentVolumeClaim:
|
||
claimName: manna-pvc
|
||
containers:
|
||
- name: write-manna
|
||
image: exo.container-registry.com/the-recognition-protocol/korban-processor:c0d0555a
|
||
imagePullPolicy: IfNotPresent
|
||
envFrom:
|
||
- configMapRef:
|
||
name: shorashim-config
|
||
volumeMounts:
|
||
- name: manna
|
||
mountPath: /var/lib/bikoret/manna
|
||
resources:
|
||
requests: { cpu: "20m", memory: "32Mi" }
|
||
limits: { cpu: "200m", memory: "128Mi" }
|
||
command: ["python3", "-c"]
|
||
args:
|
||
- |
|
||
import json, os, datetime as dt
|
||
now = dt.datetime.now(dt.timezone.utc)
|
||
today = now.date()
|
||
friday = today.weekday() == 4
|
||
units = int(os.environ.get("MANNA_COMPUTE_UNITS", "1000"))
|
||
provision = {
|
||
"compute_units": units * (2 if friday else 1),
|
||
"memory_mb": int(os.environ.get("MANNA_MEMORY_MB", "32768")),
|
||
"api_calls_external": int(os.environ.get("MANNA_API_CALLS_EXTERNAL", "500")) * (2 if friday else 1),
|
||
"storage_delta_mb": int(os.environ.get("MANNA_STORAGE_DELTA_MB", "100")),
|
||
"sanhedrin_panels_max": int(os.environ.get("MANNA_SANHEDRIN_PANELS_MAX", "5")),
|
||
}
|
||
# Regular: through the day. Double (Friday): through Shabbat, to Sunday 00:30Z.
|
||
expires = (dt.datetime.combine(today + dt.timedelta(days=2), dt.time(0, 30), dt.timezone.utc)
|
||
if friday else dt.datetime.combine(today + dt.timedelta(days=1), dt.time(0, 30), dt.timezone.utc))
|
||
doc = {
|
||
"date": today.isoformat(),
|
||
"provision": provision,
|
||
"type": "double" if friday else "regular",
|
||
"expires_at": expires.strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||
"written_by": "manna-cronjob",
|
||
"written_at": now.strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||
}
|
||
path = os.environ.get("MANNA_PROVISION_FILE", "/var/lib/bikoret/manna/today.json")
|
||
tmp = path + ".tmp"
|
||
with open(tmp, "w") as fh:
|
||
json.dump(doc, fh, indent=2)
|
||
os.replace(tmp, path)
|
||
print("manna written", doc["date"], doc["type"], "expires", doc["expires_at"])
|