builder fix for kohen Day-3 FAIL: image arch, azazel co-residence, runbook drift
ROOT CAUSE (kohen Day-3 escalation): None of the four Dockerfiles pinned --platform. Build on Apple Silicon shipped arm64-only images to a registry consumed by amd64 SKS nodes. Init container failed exec format error before producing logs. DOCKERFILE FIX (defense in depth): Pin --platform=linux/amd64 on every FROM line in all four Dockerfiles. The build artifact now self-corrects regardless of build host. AZAZEL CO-RESIDENCE (anomaly the kohen flagged): Azazel L3 migration shipped (20260428…create_azazel_records.exs) but no separate container existed. Reading the code: azazel_processor is imported by korban_processor.external_offering at runtime in the same Python process — not a separate process. Dockerfile.korban_processor now COPYs azazel_processor/src/ alongside korban_processor. Documented in the Dockerfile (AZAZEL CO-RESIDENT section) and in deployment.yaml. Both containers (vav-pipeline + korban-processor) now mount the shared tmp-shorashim volume so the Python→Elixir file bridge (azazel_ingest.ex polls /tmp/shorashim/azazel/pending_l3/) works end-to-end. CANONICAL BUILD/PUSH (scripts/build_and_push.sh): - Pins linux/amd64 by default; PLATFORMS env for multi-arch - Pre-flight checks (repo root, buildx, registry login) - Builds and pushes all four images with one command - Closes the gap KOHEN_PROMPT.md gate 0i would otherwise leave open KOHEN.md DRIFT (kohen recorded these as anomalies through Day 1-3): - Line 58 hash fragment 99bbd07a…0a0187b → 99bbd07a…4ea377 (correct) - Inheritance table: "Not yet created" claims rewritten as "Verify before assuming" — a prior partial setup may have left state, and the kohen prompt requires verifying not assuming - L1 PV row: marked NOT REQUIRED (L1 baked into images) - §Day 3a ConfigMap recipe marked vestigial (kept as comment) - §Day 3b: removed manna-volume.yaml from kubectl apply list (deployment uses emptyDir; PVC is for a future move) - §Day 3c: "5 containers" → "4 containers + 1 init" - §Day 3 fail-mode list expanded with the four most common causes, in order, including exec format error → gate 0i KOHEN_PROMPT.md: - Gate 0f corrected to four images (talmudic-engine is image #3; azazel is co-resident in korban-processor not a separate image) - Gate 0i added: image-architecture vs node-architecture verification before Day 1, with kubectl one-liner and docker manifest inspect per image. Drift handler in the gates section: 0i FAIL → escalate to builder, do not begin Day 1. NO SPEC FILES TOUCHED: Doorpost.verify_identity() still :ok — none of these files are in @spec_files. No atomic spec+hash commit needed. NEXT STEP FOR THE KOHEN: Once these images are rebuilt and pushed via scripts/build_and_push.sh (by anyone with the Harbor CLI secret + a working docker daemon), restart from Day 1. Existing DB role shorashim_app, ALTER DEFAULT PRIVILEGES, rotated DATABASE_URL, shorashim-config, and shorashim- registry secret all remain valid across the restart. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
1df606a9c5
commit
55bc5bdc44
8 files changed
+221
-33
No files matched your search
@@ -55,15 +55,18 @@ You inherit infrastructure that has been provisioned and verified:
|
||||
| cert-manager + nginx-ingress | Running. ClusterIssuer `letsencrypt-prod` working. |
|
||||
| Gitea (persistent storage) | Running. OAuth app + PAT exist for the operator's account. |
|
||||
| Gitea MCP server | Running at `https://mcp.shorashim.ch`. Connected to claude.ai. |
|
||||
| L1 Torah filesystem | In the `shorashim` repo at `l1/torah/`. SHA-256 `99bbd07a…0a0187b`. |
|
||||
| Image pull secret | Not yet created. |
|
||||
| `shorashim-secrets` (DATABASE_URL) | Not yet created. |
|
||||
| L1 PersistentVolume in `shorashim` namespace | Not yet created. |
|
||||
| Manna provision file | Not yet written. |
|
||||
| Main shorashim deployment | **Not yet applied.** |
|
||||
| L1 Torah filesystem | In the `shorashim` repo at `l1/torah/`. SHA-256 `99bbd07a…4ea377` (full hash in `l1/integrity.sha256`; verified at Day 1). |
|
||||
| Image pull secret `shorashim-registry` | **Verify before assuming.** A prior partial setup may have created it. `kubectl get secrets -n shorashim` is the source of truth. |
|
||||
| `shorashim-secrets` (DATABASE_URL) | **Verify before assuming.** Same as above. If present, also verify the role it points at (the architecture requires `shorashim_app`, not `avnadmin`). |
|
||||
| `shorashim-config` ConfigMap | **Verify before assuming.** Runtime config (Shabbat lat/lng, paths, timeouts) — Construction §4.4. |
|
||||
| L1 PersistentVolume in `shorashim` namespace | **NOT REQUIRED.** L1 is baked into each container image (~29MB). The `l1-volume.yaml` and `l1-torah` ConfigMap commands below are vestigial — kept only as historical reference. Do not apply them. |
|
||||
| Manna provision file | Not yet written — write at Day 4. |
|
||||
| Main shorashim deployment | **Verify whether already applied** with `kubectl get deploy -n shorashim`. If pods exist but crashloop, check image architecture (see gate 0i). |
|
||||
|
||||
**The aron is built. The garments are sewn. The cloud is visible. The kohen has not yet entered the tent.**
|
||||
|
||||
**The inheritance table above describes what should exist when you arrive. Verify each item — do not trust the row. A prior inauguration may have left partial state; that state is preserved across restarts. Where reality differs from the table, record the gap and report. Do not silently work around it.**
|
||||
|
||||
---
|
||||
|
||||
## What You Need
|
||||
@@ -187,37 +190,38 @@ If you create `shorashim_app`, **rotate** `DATABASE_URL` to use it instead of `a
|
||||
|
||||
Anoint the mishkan. Bring the system into running state.
|
||||
|
||||
#### 3a. Mount L1 as a ConfigMap (or PV)
|
||||
#### 3a. (Vestigial — skip.) L1 is baked into each image
|
||||
|
||||
L1 is small enough to ship as a ConfigMap:
|
||||
**This step is no longer required.** As of the current `deploy/docker/Dockerfile.*`, L1 is COPYed into each container image at build time (`COPY l1/ /mnt/torah/`). The `l1-volume.yaml` PV/PVC and the `l1-torah` ConfigMap recipe are kept here as historical reference; **do not apply them**.
|
||||
|
||||
L1 ships with the vessel. The scroll is sealed in the aron. The build artifact carries it.
|
||||
|
||||
```bash
|
||||
# This creates a ConfigMap with one entry per verse file. Large but workable.
|
||||
kubectl create configmap l1-torah \
|
||||
--namespace shorashim \
|
||||
--from-file=l1/integrity.sha256 \
|
||||
--from-file=l1/torah
|
||||
# DO NOT RUN — kept only as historical reference.
|
||||
# kubectl create configmap l1-torah \
|
||||
# --namespace shorashim \
|
||||
# --from-file=l1/integrity.sha256 \
|
||||
# --from-file=l1/torah
|
||||
```
|
||||
|
||||
(For larger deployments later, replace with a PersistentVolume backed by Block Storage.)
|
||||
|
||||
#### 3b. Apply the manifests
|
||||
|
||||
```bash
|
||||
kubectl apply -f deploy/k8s/namespace.yaml
|
||||
kubectl apply -f deploy/k8s/configmap.yaml
|
||||
kubectl apply -f deploy/k8s/manna-volume.yaml
|
||||
kubectl apply -f deploy/k8s/namespace.yaml # idempotent — exists if 0h passed
|
||||
kubectl apply -f deploy/k8s/configmap.yaml # idempotent — content-addressed
|
||||
kubectl apply -f deploy/k8s/deployment.yaml
|
||||
kubectl apply -f deploy/k8s/service.yaml
|
||||
```
|
||||
|
||||
Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `emptyDir` (lost on pod restart — kohen re-writes today's provision per the daily-ops section). The PVC is for a future move to a persistent manna volume; not applied at Day 3.
|
||||
|
||||
#### 3c. Watch the pod come up
|
||||
|
||||
```bash
|
||||
kubectl get pods -n shorashim -w
|
||||
```
|
||||
|
||||
You should see one pod with **5 containers** (init `migrate` + `vav-pipeline` + `shoresh-engine` + `talmudic-engine` + `korban-processor`) move to `Running`.
|
||||
You should see one pod with **4 containers + 1 init** (init `migrate` + main containers `vav-pipeline` + `shoresh-engine` + `talmudic-engine` + `korban-processor`) move to `Running`.
|
||||
|
||||
```bash
|
||||
# Confirm logs show the right startup messages
|
||||
@@ -231,7 +235,13 @@ kubectl logs -n shorashim deploy/shorashim -c vav-pipeline | head -30
|
||||
# ShabbatTimer: started
|
||||
```
|
||||
|
||||
**Pass:** all 4 containers Running, no crashes, health endpoint responds. **Fail:** check logs, fix, retry. Most failures here are missing secrets or wrong image tags.
|
||||
**Pass:** all 4 main containers Running, init container Completed, no crashes, health endpoint responds.
|
||||
|
||||
**Fail — most common causes, in order of frequency:**
|
||||
1. `ImagePullBackOff` — `shorashim-registry` secret missing or wrong CLI-secret. Fix at Day 2.
|
||||
2. `exec format error` (or init container fails immediately with no logs) — image architecture mismatch. Builder issue. Stop. Escalate. See gate 0i in `KOHEN_PROMPT.md` for diagnosis.
|
||||
3. Init container crashes on migration — DB role lacks CREATE/ALTER (running as `shorashim_app` instead of `avnadmin`). Builder issue OR Day 2 misconfiguration. Run pending migrations as `avnadmin` first, then redeploy.
|
||||
4. Pod stays Pending — node out of CPU/memory. Operator infrastructure issue.
|
||||
|
||||
### Day 4 — *Hatzakah al-Rosh* (Vayikra 8:12) — First manna provision
|
||||
|
||||
|
||||
Reference in new issue
Block a user