builder fix for kohen Day-3 FAIL: image arch, azazel co-residence, runbook drift
ROOT CAUSE (kohen Day-3 escalation): None of the four Dockerfiles pinned --platform. Build on Apple Silicon shipped arm64-only images to a registry consumed by amd64 SKS nodes. Init container failed exec format error before producing logs. DOCKERFILE FIX (defense in depth): Pin --platform=linux/amd64 on every FROM line in all four Dockerfiles. The build artifact now self-corrects regardless of build host. AZAZEL CO-RESIDENCE (anomaly the kohen flagged): Azazel L3 migration shipped (20260428…create_azazel_records.exs) but no separate container existed. Reading the code: azazel_processor is imported by korban_processor.external_offering at runtime in the same Python process — not a separate process. Dockerfile.korban_processor now COPYs azazel_processor/src/ alongside korban_processor. Documented in the Dockerfile (AZAZEL CO-RESIDENT section) and in deployment.yaml. Both containers (vav-pipeline + korban-processor) now mount the shared tmp-shorashim volume so the Python→Elixir file bridge (azazel_ingest.ex polls /tmp/shorashim/azazel/pending_l3/) works end-to-end. CANONICAL BUILD/PUSH (scripts/build_and_push.sh): - Pins linux/amd64 by default; PLATFORMS env for multi-arch - Pre-flight checks (repo root, buildx, registry login) - Builds and pushes all four images with one command - Closes the gap KOHEN_PROMPT.md gate 0i would otherwise leave open KOHEN.md DRIFT (kohen recorded these as anomalies through Day 1-3): - Line 58 hash fragment 99bbd07a…0a0187b → 99bbd07a…4ea377 (correct) - Inheritance table: "Not yet created" claims rewritten as "Verify before assuming" — a prior partial setup may have left state, and the kohen prompt requires verifying not assuming - L1 PV row: marked NOT REQUIRED (L1 baked into images) - §Day 3a ConfigMap recipe marked vestigial (kept as comment) - §Day 3b: removed manna-volume.yaml from kubectl apply list (deployment uses emptyDir; PVC is for a future move) - §Day 3c: "5 containers" → "4 containers + 1 init" - §Day 3 fail-mode list expanded with the four most common causes, in order, including exec format error → gate 0i KOHEN_PROMPT.md: - Gate 0f corrected to four images (talmudic-engine is image #3; azazel is co-resident in korban-processor not a separate image) - Gate 0i added: image-architecture vs node-architecture verification before Day 1, with kubectl one-liner and docker manifest inspect per image. Drift handler in the gates section: 0i FAIL → escalate to builder, do not begin Day 1. NO SPEC FILES TOUCHED: Doorpost.verify_identity() still :ok — none of these files are in @spec_files. No atomic spec+hash commit needed. NEXT STEP FOR THE KOHEN: Once these images are rebuilt and pushed via scripts/build_and_push.sh (by anyone with the Harbor CLI secret + a working docker daemon), restart from Day 1. Existing DB role shorashim_app, ALTER DEFAULT PRIVILEGES, rotated DATABASE_URL, shorashim-config, and shorashim- registry secret all remain valid across the restart. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
1df606a9c5
commit
55bc5bdc44
8 files changed
+221
-33
No files matched your search
+32
-4
@@ -87,10 +87,15 @@ mix run -e 'IO.inspect VavPipeline.Doorpost.verify_identity()'
|
||||
# Expect: :ok
|
||||
# If {:error, ...}: a spec file has drifted. STOP. The builder, not you, addresses spec drift.
|
||||
|
||||
# 0f. Container images present in the registry
|
||||
exo iam api-key list # confirm registry credentials work
|
||||
# Then per the runbook: confirm vav_pipeline, korban_processor, azazel_processor, shoresh-engine
|
||||
# images exist tagged :latest in the-recognition-protocol registry.
|
||||
# 0f. Container images present in the registry, AT THE RIGHT ARCHITECTURE.
|
||||
# Four images, all tagged :latest in exo.container-registry.com/the-recognition-protocol/:
|
||||
# - vav-pipeline
|
||||
# - shoresh-engine
|
||||
# - talmudic-engine
|
||||
# - korban-processor (azazel_processor is co-resident inside this image — not separate)
|
||||
# Registry credentials check:
|
||||
exo iam api-key list # owner key + any service keys should be present
|
||||
# Material image-presence verification happens at Day 3 (pod scheduling).
|
||||
|
||||
# 0g. cert-manager + ingress controller present
|
||||
kubectl get pods -n cert-manager
|
||||
@@ -98,6 +103,27 @@ kubectl get pods -n ingress-nginx # or traefik, whichever was installed
|
||||
|
||||
# 0h. Namespace
|
||||
kubectl get ns shorashim || kubectl create ns shorashim
|
||||
|
||||
# 0i. Image architecture matches node architecture.
|
||||
# Cluster nodes are linux/amd64. If the build was run on Apple Silicon
|
||||
# without --platform=linux/amd64, the images are arm64-only and the
|
||||
# deployment will fail at Day 3 with "exec format error" — usually
|
||||
# before any logs are produced, because the init container's binary
|
||||
# cannot even start. This gate catches it before Day 1 begins.
|
||||
NODE_ARCH=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}')
|
||||
echo "Node arch: $NODE_ARCH (expected: amd64)"
|
||||
# Verify each image manifest's architecture matches. Requires docker login
|
||||
# to the Harbor registry first (CLI secret from Harbor User Profile):
|
||||
docker login exo.container-registry.com
|
||||
for img in vav-pipeline shoresh-engine talmudic-engine korban-processor; do
|
||||
arch=$(docker manifest inspect "exo.container-registry.com/the-recognition-protocol/$img:latest" \
|
||||
| jq -r '.architecture // .manifests[].platform.architecture' \
|
||||
| sort -u | tr '\n' ',')
|
||||
echo "$img -> $arch (expected: contains amd64)"
|
||||
done
|
||||
# Pass: every image's architectures contain amd64.
|
||||
# Fail: builder action required — see scripts/build_and_push.sh for the
|
||||
# canonical buildx procedure that emits amd64 (or multi-arch) artifacts.
|
||||
```
|
||||
|
||||
**Gates:** every check passes, OR the gap is recorded and an explicit decision is made about it. No silent assumption.
|
||||
@@ -106,6 +132,8 @@ If the gate is `0e Doorpost FAIL`: the spec set has drifted from the witnessed s
|
||||
|
||||
If the gate is `0d L1 FAIL`: the Torah text differs from the verified hash. This is a category-1 event. Halt all activity. Report immediately.
|
||||
|
||||
If the gate is `0i image-arch FAIL`: the build artifact's architecture does not match the cluster's nodes. This is **not** your problem to fix. Report to the transmitter; the builder rebuilds with `scripts/build_and_push.sh` (which pins `--platform=linux/amd64`). Without this fix the deployment cannot run — the kernel will refuse to exec the binaries. Do not begin Day 1.
|
||||
|
||||
---
|
||||
|
||||
## The Seven Days
|
||||
|
||||
Reference in new issue
Block a user