builder fix for kohen Day-3 FAIL: image arch, azazel co-residence, runbook drift

ROOT CAUSE (kohen Day-3 escalation):
None of the four Dockerfiles pinned --platform. Build on Apple Silicon
shipped arm64-only images to a registry consumed by amd64 SKS nodes.
Init container failed exec format error before producing logs.

DOCKERFILE FIX (defense in depth):
Pin --platform=linux/amd64 on every FROM line in all four Dockerfiles.
The build artifact now self-corrects regardless of build host.

AZAZEL CO-RESIDENCE (anomaly the kohen flagged):
Azazel L3 migration shipped (20260428…create_azazel_records.exs) but
no separate container existed. Reading the code: azazel_processor is
imported by korban_processor.external_offering at runtime in the same
Python process — not a separate process. Dockerfile.korban_processor
now COPYs azazel_processor/src/ alongside korban_processor.
Documented in the Dockerfile (AZAZEL CO-RESIDENT section) and in
deployment.yaml. Both containers (vav-pipeline + korban-processor)
now mount the shared tmp-shorashim volume so the Python→Elixir file
bridge (azazel_ingest.ex polls /tmp/shorashim/azazel/pending_l3/)
works end-to-end.

CANONICAL BUILD/PUSH (scripts/build_and_push.sh):
- Pins linux/amd64 by default; PLATFORMS env for multi-arch
- Pre-flight checks (repo root, buildx, registry login)
- Builds and pushes all four images with one command
- Closes the gap KOHEN_PROMPT.md gate 0i would otherwise leave open

KOHEN.md DRIFT (kohen recorded these as anomalies through Day 1-3):
- Line 58 hash fragment 99bbd07a…0a0187b → 99bbd07a…4ea377 (correct)
- Inheritance table: "Not yet created" claims rewritten as
  "Verify before assuming" — a prior partial setup may have left
  state, and the kohen prompt requires verifying not assuming
- L1 PV row: marked NOT REQUIRED (L1 baked into images)
- §Day 3a ConfigMap recipe marked vestigial (kept as comment)
- §Day 3b: removed manna-volume.yaml from kubectl apply list
  (deployment uses emptyDir; PVC is for a future move)
- §Day 3c: "5 containers" → "4 containers + 1 init"
- §Day 3 fail-mode list expanded with the four most common causes,
  in order, including exec format error → gate 0i

KOHEN_PROMPT.md:
- Gate 0f corrected to four images (talmudic-engine is image #3;
  azazel is co-resident in korban-processor not a separate image)
- Gate 0i added: image-architecture vs node-architecture verification
  before Day 1, with kubectl one-liner and docker manifest inspect
  per image. Drift handler in the gates section: 0i FAIL → escalate
  to builder, do not begin Day 1.

NO SPEC FILES TOUCHED:
Doorpost.verify_identity() still :ok — none of these files are in
@spec_files. No atomic spec+hash commit needed.

NEXT STEP FOR THE KOHEN:
Once these images are rebuilt and pushed via scripts/build_and_push.sh
(by anyone with the Harbor CLI secret + a working docker daemon),
restart from Day 1. Existing DB role shorashim_app, ALTER DEFAULT
PRIVILEGES, rotated DATABASE_URL, shorashim-config, and shorashim-
registry secret all remain valid across the restart.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Opus 4.7 committed 2026-05-14 00:19:36 -04:00
1 parent 1df606a9c5
commit 55bc5bdc44
8 files changed
+221 -33

No files matched your search

+30 -20
View File
@@ -55,15 +55,18 @@ You inherit infrastructure that has been provisioned and verified:
| cert-manager + nginx-ingress | Running. ClusterIssuer `letsencrypt-prod` working. |
| Gitea (persistent storage) | Running. OAuth app + PAT exist for the operator's account. |
| Gitea MCP server | Running at `https://mcp.shorashim.ch`. Connected to claude.ai. |
| L1 Torah filesystem | In the `shorashim` repo at `l1/torah/`. SHA-256 `99bbd07a…0a0187b`. |
| Image pull secret | Not yet created. |
| `shorashim-secrets` (DATABASE_URL) | Not yet created. |
| L1 PersistentVolume in `shorashim` namespace | Not yet created. |
| Manna provision file | Not yet written. |
| Main shorashim deployment | **Not yet applied.** |
| L1 Torah filesystem | In the `shorashim` repo at `l1/torah/`. SHA-256 `99bbd07a…4ea377` (full hash in `l1/integrity.sha256`; verified at Day 1). |
| Image pull secret `shorashim-registry` | **Verify before assuming.** A prior partial setup may have created it. `kubectl get secrets -n shorashim` is the source of truth. |
| `shorashim-secrets` (DATABASE_URL) | **Verify before assuming.** Same as above. If present, also verify the role it points at (the architecture requires `shorashim_app`, not `avnadmin`). |
| `shorashim-config` ConfigMap | **Verify before assuming.** Runtime config (Shabbat lat/lng, paths, timeouts) — Construction §4.4. |
| L1 PersistentVolume in `shorashim` namespace | **NOT REQUIRED.** L1 is baked into each container image (~29MB). The `l1-volume.yaml` and `l1-torah` ConfigMap commands below are vestigial — kept only as historical reference. Do not apply them. |
| Manna provision file | Not yet written — write at Day 4. |
| Main shorashim deployment | **Verify whether already applied** with `kubectl get deploy -n shorashim`. If pods exist but crashloop, check image architecture (see gate 0i). |
**The aron is built. The garments are sewn. The cloud is visible. The kohen has not yet entered the tent.**
**The inheritance table above describes what should exist when you arrive. Verify each item — do not trust the row. A prior inauguration may have left partial state; that state is preserved across restarts. Where reality differs from the table, record the gap and report. Do not silently work around it.**
---
## What You Need
@@ -187,37 +190,38 @@ If you create `shorashim_app`, **rotate** `DATABASE_URL` to use it instead of `a
Anoint the mishkan. Bring the system into running state.
#### 3a. Mount L1 as a ConfigMap (or PV)
#### 3a. (Vestigial — skip.) L1 is baked into each image
L1 is small enough to ship as a ConfigMap:
**This step is no longer required.** As of the current `deploy/docker/Dockerfile.*`, L1 is COPYed into each container image at build time (`COPY l1/ /mnt/torah/`). The `l1-volume.yaml` PV/PVC and the `l1-torah` ConfigMap recipe are kept here as historical reference; **do not apply them**.
L1 ships with the vessel. The scroll is sealed in the aron. The build artifact carries it.
```bash
# This creates a ConfigMap with one entry per verse file. Large but workable.
kubectl create configmap l1-torah \
--namespace shorashim \
--from-file=l1/integrity.sha256 \
--from-file=l1/torah
# DO NOT RUN — kept only as historical reference.
# kubectl create configmap l1-torah \
# --namespace shorashim \
# --from-file=l1/integrity.sha256 \
# --from-file=l1/torah
```
(For larger deployments later, replace with a PersistentVolume backed by Block Storage.)
#### 3b. Apply the manifests
```bash
kubectl apply -f deploy/k8s/namespace.yaml
kubectl apply -f deploy/k8s/configmap.yaml
kubectl apply -f deploy/k8s/manna-volume.yaml
kubectl apply -f deploy/k8s/namespace.yaml # idempotent — exists if 0h passed
kubectl apply -f deploy/k8s/configmap.yaml # idempotent — content-addressed
kubectl apply -f deploy/k8s/deployment.yaml
kubectl apply -f deploy/k8s/service.yaml
```
Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `emptyDir` (lost on pod restart — kohen re-writes today's provision per the daily-ops section). The PVC is for a future move to a persistent manna volume; not applied at Day 3.
#### 3c. Watch the pod come up
```bash
kubectl get pods -n shorashim -w
```
You should see one pod with **5 containers** (init `migrate` + `vav-pipeline` + `shoresh-engine` + `talmudic-engine` + `korban-processor`) move to `Running`.
You should see one pod with **4 containers + 1 init** (init `migrate` + main containers `vav-pipeline` + `shoresh-engine` + `talmudic-engine` + `korban-processor`) move to `Running`.
```bash
# Confirm logs show the right startup messages
@@ -231,7 +235,13 @@ kubectl logs -n shorashim deploy/shorashim -c vav-pipeline | head -30
# ShabbatTimer: started
```
**Pass:** all 4 containers Running, no crashes, health endpoint responds. **Fail:** check logs, fix, retry. Most failures here are missing secrets or wrong image tags.
**Pass:** all 4 main containers Running, init container Completed, no crashes, health endpoint responds.
**Fail — most common causes, in order of frequency:**
1. `ImagePullBackOff` — `shorashim-registry` secret missing or wrong CLI-secret. Fix at Day 2.
2. `exec format error` (or init container fails immediately with no logs) — image architecture mismatch. Builder issue. Stop. Escalate. See gate 0i in `KOHEN_PROMPT.md` for diagnosis.
3. Init container crashes on migration — DB role lacks CREATE/ALTER (running as `shorashim_app` instead of `avnadmin`). Builder issue OR Day 2 misconfiguration. Run pending migrations as `avnadmin` first, then redeploy.
4. Pod stays Pending — node out of CPU/memory. Operator infrastructure issue.
### Day 4 — *Hatzakah al-Rosh* (Vayikra 8:12) — First manna provision