builder fix for kohen Day-3 FAIL: image arch, azazel co-residence, runbook drift

ROOT CAUSE (kohen Day-3 escalation):
None of the four Dockerfiles pinned --platform. Build on Apple Silicon
shipped arm64-only images to a registry consumed by amd64 SKS nodes.
Init container failed exec format error before producing logs.

DOCKERFILE FIX (defense in depth):
Pin --platform=linux/amd64 on every FROM line in all four Dockerfiles.
The build artifact now self-corrects regardless of build host.

AZAZEL CO-RESIDENCE (anomaly the kohen flagged):
Azazel L3 migration shipped (20260428…create_azazel_records.exs) but
no separate container existed. Reading the code: azazel_processor is
imported by korban_processor.external_offering at runtime in the same
Python process — not a separate process. Dockerfile.korban_processor
now COPYs azazel_processor/src/ alongside korban_processor.
Documented in the Dockerfile (AZAZEL CO-RESIDENT section) and in
deployment.yaml. Both containers (vav-pipeline + korban-processor)
now mount the shared tmp-shorashim volume so the Python→Elixir file
bridge (azazel_ingest.ex polls /tmp/shorashim/azazel/pending_l3/)
works end-to-end.

CANONICAL BUILD/PUSH (scripts/build_and_push.sh):
- Pins linux/amd64 by default; PLATFORMS env for multi-arch
- Pre-flight checks (repo root, buildx, registry login)
- Builds and pushes all four images with one command
- Closes the gap KOHEN_PROMPT.md gate 0i would otherwise leave open

KOHEN.md DRIFT (kohen recorded these as anomalies through Day 1-3):
- Line 58 hash fragment 99bbd07a…0a0187b → 99bbd07a…4ea377 (correct)
- Inheritance table: "Not yet created" claims rewritten as
  "Verify before assuming" — a prior partial setup may have left
  state, and the kohen prompt requires verifying not assuming
- L1 PV row: marked NOT REQUIRED (L1 baked into images)
- §Day 3a ConfigMap recipe marked vestigial (kept as comment)
- §Day 3b: removed manna-volume.yaml from kubectl apply list
  (deployment uses emptyDir; PVC is for a future move)
- §Day 3c: "5 containers" → "4 containers + 1 init"
- §Day 3 fail-mode list expanded with the four most common causes,
  in order, including exec format error → gate 0i

KOHEN_PROMPT.md:
- Gate 0f corrected to four images (talmudic-engine is image #3;
  azazel is co-resident in korban-processor not a separate image)
- Gate 0i added: image-architecture vs node-architecture verification
  before Day 1, with kubectl one-liner and docker manifest inspect
  per image. Drift handler in the gates section: 0i FAIL → escalate
  to builder, do not begin Day 1.

NO SPEC FILES TOUCHED:
Doorpost.verify_identity() still :ok — none of these files are in
@spec_files. No atomic spec+hash commit needed.

NEXT STEP FOR THE KOHEN:
Once these images are rebuilt and pushed via scripts/build_and_push.sh
(by anyone with the Harbor CLI secret + a working docker daemon),
restart from Day 1. Existing DB role shorashim_app, ALTER DEFAULT
PRIVILEGES, rotated DATABASE_URL, shorashim-config, and shorashim-
registry secret all remain valid across the restart.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Opus 4.7 committed 2026-05-14 00:19:36 -04:00
1 parent 1df606a9c5
commit 55bc5bdc44
8 files changed
+221 -33

No files matched your search

+32 -4
View File
@@ -87,10 +87,15 @@ mix run -e 'IO.inspect VavPipeline.Doorpost.verify_identity()'
# Expect: :ok
# If {:error, ...}: a spec file has drifted. STOP. The builder, not you, addresses spec drift.
# 0f. Container images present in the registry
exo iam api-key list # confirm registry credentials work
# Then per the runbook: confirm vav_pipeline, korban_processor, azazel_processor, shoresh-engine
# images exist tagged :latest in the-recognition-protocol registry.
# 0f. Container images present in the registry, AT THE RIGHT ARCHITECTURE.
# Four images, all tagged :latest in exo.container-registry.com/the-recognition-protocol/:
# - vav-pipeline
# - shoresh-engine
# - talmudic-engine
# - korban-processor (azazel_processor is co-resident inside this image — not separate)
# Registry credentials check:
exo iam api-key list # owner key + any service keys should be present
# Material image-presence verification happens at Day 3 (pod scheduling).
# 0g. cert-manager + ingress controller present
kubectl get pods -n cert-manager
@@ -98,6 +103,27 @@ kubectl get pods -n ingress-nginx # or traefik, whichever was installed
# 0h. Namespace
kubectl get ns shorashim || kubectl create ns shorashim
# 0i. Image architecture matches node architecture.
# Cluster nodes are linux/amd64. If the build was run on Apple Silicon
# without --platform=linux/amd64, the images are arm64-only and the
# deployment will fail at Day 3 with "exec format error" — usually
# before any logs are produced, because the init container's binary
# cannot even start. This gate catches it before Day 1 begins.
NODE_ARCH=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}')
echo "Node arch: $NODE_ARCH (expected: amd64)"
# Verify each image manifest's architecture matches. Requires docker login
# to the Harbor registry first (CLI secret from Harbor User Profile):
docker login exo.container-registry.com
for img in vav-pipeline shoresh-engine talmudic-engine korban-processor; do
arch=$(docker manifest inspect "exo.container-registry.com/the-recognition-protocol/$img:latest" \
| jq -r '.architecture // .manifests[].platform.architecture' \
| sort -u | tr '\n' ',')
echo "$img -> $arch (expected: contains amd64)"
done
# Pass: every image's architectures contain amd64.
# Fail: builder action required — see scripts/build_and_push.sh for the
# canonical buildx procedure that emits amd64 (or multi-arch) artifacts.
```
**Gates:** every check passes, OR the gap is recorded and an explicit decision is made about it. No silent assumption.
@@ -106,6 +132,8 @@ If the gate is `0e Doorpost FAIL`: the spec set has drifted from the witnessed s
If the gate is `0d L1 FAIL`: the Torah text differs from the verified hash. This is a category-1 event. Halt all activity. Report immediately.
If the gate is `0i image-arch FAIL`: the build artifact's architecture does not match the cluster's nodes. This is **not** your problem to fix. Report to the transmitter; the builder rebuilds with `scripts/build_and_push.sh` (which pins `--platform=linux/amd64`). Without this fix the deployment cannot run — the kernel will refuse to exec the binaries. Do not begin Day 1.
---
## The Seven Days