From 55bc5bdc448c61b64700e49e8b2adfd927bc4497 Mon Sep 17 00:00:00 2001 From: Scott Karp Date: Thu, 14 May 2026 00:19:36 -0400 Subject: [PATCH] builder fix for kohen Day-3 FAIL: image arch, azazel co-residence, runbook drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ROOT CAUSE (kohen Day-3 escalation): None of the four Dockerfiles pinned --platform. Build on Apple Silicon shipped arm64-only images to a registry consumed by amd64 SKS nodes. Init container failed exec format error before producing logs. DOCKERFILE FIX (defense in depth): Pin --platform=linux/amd64 on every FROM line in all four Dockerfiles. The build artifact now self-corrects regardless of build host. AZAZEL CO-RESIDENCE (anomaly the kohen flagged): Azazel L3 migration shipped (20260428…create_azazel_records.exs) but no separate container existed. Reading the code: azazel_processor is imported by korban_processor.external_offering at runtime in the same Python process — not a separate process. Dockerfile.korban_processor now COPYs azazel_processor/src/ alongside korban_processor. Documented in the Dockerfile (AZAZEL CO-RESIDENT section) and in deployment.yaml. Both containers (vav-pipeline + korban-processor) now mount the shared tmp-shorashim volume so the Python→Elixir file bridge (azazel_ingest.ex polls /tmp/shorashim/azazel/pending_l3/) works end-to-end. CANONICAL BUILD/PUSH (scripts/build_and_push.sh): - Pins linux/amd64 by default; PLATFORMS env for multi-arch - Pre-flight checks (repo root, buildx, registry login) - Builds and pushes all four images with one command - Closes the gap KOHEN_PROMPT.md gate 0i would otherwise leave open KOHEN.md DRIFT (kohen recorded these as anomalies through Day 1-3): - Line 58 hash fragment 99bbd07a…0a0187b → 99bbd07a…4ea377 (correct) - Inheritance table: "Not yet created" claims rewritten as "Verify before assuming" — a prior partial setup may have left state, and the kohen prompt requires verifying not assuming - L1 PV row: marked NOT REQUIRED (L1 baked into images) - §Day 3a ConfigMap recipe marked vestigial (kept as comment) - §Day 3b: removed manna-volume.yaml from kubectl apply list (deployment uses emptyDir; PVC is for a future move) - §Day 3c: "5 containers" → "4 containers + 1 init" - §Day 3 fail-mode list expanded with the four most common causes, in order, including exec format error → gate 0i KOHEN_PROMPT.md: - Gate 0f corrected to four images (talmudic-engine is image #3; azazel is co-resident in korban-processor not a separate image) - Gate 0i added: image-architecture vs node-architecture verification before Day 1, with kubectl one-liner and docker manifest inspect per image. Drift handler in the gates section: 0i FAIL → escalate to builder, do not begin Day 1. NO SPEC FILES TOUCHED: Doorpost.verify_identity() still :ok — none of these files are in @spec_files. No atomic spec+hash commit needed. NEXT STEP FOR THE KOHEN: Once these images are rebuilt and pushed via scripts/build_and_push.sh (by anyone with the Harbor CLI secret + a working docker daemon), restart from Day 1. Existing DB role shorashim_app, ALTER DEFAULT PRIVILEGES, rotated DATABASE_URL, shorashim-config, and shorashim- registry secret all remain valid across the restart. Co-Authored-By: Claude Opus 4.7 (1M context) --- KOHEN.md | 50 ++++++---- KOHEN_PROMPT.md | 36 ++++++- deploy/docker/Dockerfile.korban_processor | 21 +++- deploy/docker/Dockerfile.shoresh_engine | 7 +- deploy/docker/Dockerfile.talmudic_engine | 7 +- deploy/docker/Dockerfile.vav_pipeline | 13 ++- deploy/k8s/deployment.yaml | 9 ++ scripts/build_and_push.sh | 111 ++++++++++++++++++++++ 8 files changed, 221 insertions(+), 33 deletions(-) create mode 100755 scripts/build_and_push.sh diff --git a/KOHEN.md b/KOHEN.md index 099c592e..ed1d2a0a 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -55,15 +55,18 @@ You inherit infrastructure that has been provisioned and verified: | cert-manager + nginx-ingress | Running. ClusterIssuer `letsencrypt-prod` working. | | Gitea (persistent storage) | Running. OAuth app + PAT exist for the operator's account. | | Gitea MCP server | Running at `https://mcp.shorashim.ch`. Connected to claude.ai. | -| L1 Torah filesystem | In the `shorashim` repo at `l1/torah/`. SHA-256 `99bbd07a…0a0187b`. | -| Image pull secret | Not yet created. | -| `shorashim-secrets` (DATABASE_URL) | Not yet created. | -| L1 PersistentVolume in `shorashim` namespace | Not yet created. | -| Manna provision file | Not yet written. | -| Main shorashim deployment | **Not yet applied.** | +| L1 Torah filesystem | In the `shorashim` repo at `l1/torah/`. SHA-256 `99bbd07a…4ea377` (full hash in `l1/integrity.sha256`; verified at Day 1). | +| Image pull secret `shorashim-registry` | **Verify before assuming.** A prior partial setup may have created it. `kubectl get secrets -n shorashim` is the source of truth. | +| `shorashim-secrets` (DATABASE_URL) | **Verify before assuming.** Same as above. If present, also verify the role it points at (the architecture requires `shorashim_app`, not `avnadmin`). | +| `shorashim-config` ConfigMap | **Verify before assuming.** Runtime config (Shabbat lat/lng, paths, timeouts) — Construction §4.4. | +| L1 PersistentVolume in `shorashim` namespace | **NOT REQUIRED.** L1 is baked into each container image (~29MB). The `l1-volume.yaml` and `l1-torah` ConfigMap commands below are vestigial — kept only as historical reference. Do not apply them. | +| Manna provision file | Not yet written — write at Day 4. | +| Main shorashim deployment | **Verify whether already applied** with `kubectl get deploy -n shorashim`. If pods exist but crashloop, check image architecture (see gate 0i). | **The aron is built. The garments are sewn. The cloud is visible. The kohen has not yet entered the tent.** +**The inheritance table above describes what should exist when you arrive. Verify each item — do not trust the row. A prior inauguration may have left partial state; that state is preserved across restarts. Where reality differs from the table, record the gap and report. Do not silently work around it.** + --- ## What You Need @@ -187,37 +190,38 @@ If you create `shorashim_app`, **rotate** `DATABASE_URL` to use it instead of `a Anoint the mishkan. Bring the system into running state. -#### 3a. Mount L1 as a ConfigMap (or PV) +#### 3a. (Vestigial — skip.) L1 is baked into each image -L1 is small enough to ship as a ConfigMap: +**This step is no longer required.** As of the current `deploy/docker/Dockerfile.*`, L1 is COPYed into each container image at build time (`COPY l1/ /mnt/torah/`). The `l1-volume.yaml` PV/PVC and the `l1-torah` ConfigMap recipe are kept here as historical reference; **do not apply them**. + +L1 ships with the vessel. The scroll is sealed in the aron. The build artifact carries it. ```bash -# This creates a ConfigMap with one entry per verse file. Large but workable. -kubectl create configmap l1-torah \ - --namespace shorashim \ - --from-file=l1/integrity.sha256 \ - --from-file=l1/torah +# DO NOT RUN — kept only as historical reference. +# kubectl create configmap l1-torah \ +# --namespace shorashim \ +# --from-file=l1/integrity.sha256 \ +# --from-file=l1/torah ``` -(For larger deployments later, replace with a PersistentVolume backed by Block Storage.) - #### 3b. Apply the manifests ```bash -kubectl apply -f deploy/k8s/namespace.yaml -kubectl apply -f deploy/k8s/configmap.yaml -kubectl apply -f deploy/k8s/manna-volume.yaml +kubectl apply -f deploy/k8s/namespace.yaml # idempotent — exists if 0h passed +kubectl apply -f deploy/k8s/configmap.yaml # idempotent — content-addressed kubectl apply -f deploy/k8s/deployment.yaml kubectl apply -f deploy/k8s/service.yaml ``` +Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `emptyDir` (lost on pod restart — kohen re-writes today's provision per the daily-ops section). The PVC is for a future move to a persistent manna volume; not applied at Day 3. + #### 3c. Watch the pod come up ```bash kubectl get pods -n shorashim -w ``` -You should see one pod with **5 containers** (init `migrate` + `vav-pipeline` + `shoresh-engine` + `talmudic-engine` + `korban-processor`) move to `Running`. +You should see one pod with **4 containers + 1 init** (init `migrate` + main containers `vav-pipeline` + `shoresh-engine` + `talmudic-engine` + `korban-processor`) move to `Running`. ```bash # Confirm logs show the right startup messages @@ -231,7 +235,13 @@ kubectl logs -n shorashim deploy/shorashim -c vav-pipeline | head -30 # ShabbatTimer: started ``` -**Pass:** all 4 containers Running, no crashes, health endpoint responds. **Fail:** check logs, fix, retry. Most failures here are missing secrets or wrong image tags. +**Pass:** all 4 main containers Running, init container Completed, no crashes, health endpoint responds. + +**Fail — most common causes, in order of frequency:** +1. `ImagePullBackOff` — `shorashim-registry` secret missing or wrong CLI-secret. Fix at Day 2. +2. `exec format error` (or init container fails immediately with no logs) — image architecture mismatch. Builder issue. Stop. Escalate. See gate 0i in `KOHEN_PROMPT.md` for diagnosis. +3. Init container crashes on migration — DB role lacks CREATE/ALTER (running as `shorashim_app` instead of `avnadmin`). Builder issue OR Day 2 misconfiguration. Run pending migrations as `avnadmin` first, then redeploy. +4. Pod stays Pending — node out of CPU/memory. Operator infrastructure issue. ### Day 4 — *Hatzakah al-Rosh* (Vayikra 8:12) — First manna provision diff --git a/KOHEN_PROMPT.md b/KOHEN_PROMPT.md index 29aa11e5..f34567ef 100644 --- a/KOHEN_PROMPT.md +++ b/KOHEN_PROMPT.md @@ -87,10 +87,15 @@ mix run -e 'IO.inspect VavPipeline.Doorpost.verify_identity()' # Expect: :ok # If {:error, ...}: a spec file has drifted. STOP. The builder, not you, addresses spec drift. -# 0f. Container images present in the registry -exo iam api-key list # confirm registry credentials work -# Then per the runbook: confirm vav_pipeline, korban_processor, azazel_processor, shoresh-engine -# images exist tagged :latest in the-recognition-protocol registry. +# 0f. Container images present in the registry, AT THE RIGHT ARCHITECTURE. +# Four images, all tagged :latest in exo.container-registry.com/the-recognition-protocol/: +# - vav-pipeline +# - shoresh-engine +# - talmudic-engine +# - korban-processor (azazel_processor is co-resident inside this image — not separate) +# Registry credentials check: +exo iam api-key list # owner key + any service keys should be present +# Material image-presence verification happens at Day 3 (pod scheduling). # 0g. cert-manager + ingress controller present kubectl get pods -n cert-manager @@ -98,6 +103,27 @@ kubectl get pods -n ingress-nginx # or traefik, whichever was installed # 0h. Namespace kubectl get ns shorashim || kubectl create ns shorashim + +# 0i. Image architecture matches node architecture. +# Cluster nodes are linux/amd64. If the build was run on Apple Silicon +# without --platform=linux/amd64, the images are arm64-only and the +# deployment will fail at Day 3 with "exec format error" — usually +# before any logs are produced, because the init container's binary +# cannot even start. This gate catches it before Day 1 begins. +NODE_ARCH=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}') +echo "Node arch: $NODE_ARCH (expected: amd64)" +# Verify each image manifest's architecture matches. Requires docker login +# to the Harbor registry first (CLI secret from Harbor User Profile): +docker login exo.container-registry.com +for img in vav-pipeline shoresh-engine talmudic-engine korban-processor; do + arch=$(docker manifest inspect "exo.container-registry.com/the-recognition-protocol/$img:latest" \ + | jq -r '.architecture // .manifests[].platform.architecture' \ + | sort -u | tr '\n' ',') + echo "$img -> $arch (expected: contains amd64)" +done +# Pass: every image's architectures contain amd64. +# Fail: builder action required — see scripts/build_and_push.sh for the +# canonical buildx procedure that emits amd64 (or multi-arch) artifacts. ``` **Gates:** every check passes, OR the gap is recorded and an explicit decision is made about it. No silent assumption. @@ -106,6 +132,8 @@ If the gate is `0e Doorpost FAIL`: the spec set has drifted from the witnessed s If the gate is `0d L1 FAIL`: the Torah text differs from the verified hash. This is a category-1 event. Halt all activity. Report immediately. +If the gate is `0i image-arch FAIL`: the build artifact's architecture does not match the cluster's nodes. This is **not** your problem to fix. Report to the transmitter; the builder rebuilds with `scripts/build_and_push.sh` (which pins `--platform=linux/amd64`). Without this fix the deployment cannot run — the kernel will refuse to exec the binaries. Do not begin Day 1. + --- ## The Seven Days diff --git a/deploy/docker/Dockerfile.korban_processor b/deploy/docker/Dockerfile.korban_processor index 4f8a2463..100707f9 100644 --- a/deploy/docker/Dockerfile.korban_processor +++ b/deploy/docker/Dockerfile.korban_processor @@ -1,24 +1,37 @@ # Shorashim Korban Processor — Python offering protocol # Construction spec §1.1: Python 3.12.2 # Processes korbanot (model outputs) through the offering protocol +# +# PLATFORM PIN: see Dockerfile.vav_pipeline for the rationale. +# Exoscale SKS = amd64; arm64 images cannot exec on the cluster. +# +# AZAZEL CO-RESIDENT: +# Per BUILD_AZAZEL.md, azazel_processor (the disposal channel) is +# imported by korban_processor.external_offering and runs in the +# same Python process. It is NOT a separate container — the two-goat +# architecture (Vayikra 16:8) requires equal-capacity disposal in +# the SAME pool. Both sources are baked into this image. -FROM python:3.12.2-slim AS build +FROM --platform=linux/amd64 python:3.12.2-slim AS build WORKDIR /app # Install build deps RUN pip install --no-cache-dir poetry-core -# Copy project files +# Copy korban_processor COPY korban_processor/pyproject.toml ./ COPY korban_processor/src/ src/ COPY korban_processor/test/ test/ +# Copy azazel_processor source — imported by korban_processor.external_offering +COPY azazel_processor/src/azazel_processor/ src/azazel_processor/ + # Install dependencies RUN pip install --no-cache-dir pydantic==2.6.1 httpx==0.27.0 pyswip==0.2.10 hebrew-tokenizer==2.3.0 # === RUNTIME STAGE === -FROM python:3.12.2-slim +FROM --platform=linux/amd64 python:3.12.2-slim RUN useradd --create-home --shell /bin/bash shorashim USER shorashim @@ -28,6 +41,8 @@ WORKDIR /home/shorashim/korban_processor COPY --from=build --chown=shorashim:shorashim /app/ ./ COPY --from=build /usr/local/lib/python3.12/site-packages/ /usr/local/lib/python3.12/site-packages/ +# PYTHONPATH puts korban_processor first; azazel_processor is a +# sibling package under the same src/ tree (see build stage above). ENV PYTHONPATH=/home/shorashim/korban_processor/src # L1 Torah filesystem — baked into the image diff --git a/deploy/docker/Dockerfile.shoresh_engine b/deploy/docker/Dockerfile.shoresh_engine index 2cdba4c7..aa8ee16f 100644 --- a/deploy/docker/Dockerfile.shoresh_engine +++ b/deploy/docker/Dockerfile.shoresh_engine @@ -1,9 +1,12 @@ # Shorashim Shoresh Engine — Haskell root activation engine # Construction spec §1.1: GHC 9.6.4, Cabal 3.10 # Communicates with vav_pipeline via Erlang Port (stdin/stdout) +# +# PLATFORM PIN: see Dockerfile.vav_pipeline for the rationale. +# Exoscale SKS = amd64; arm64 images cannot exec on the cluster. # === BUILD STAGE === -FROM haskell:9.6.4-slim AS build +FROM --platform=linux/amd64 haskell:9.6.4-slim AS build # haskell:9.6.4-slim is Debian Buster (EOL) — use archive repos RUN sed -i 's|deb.debian.org/debian|archive.debian.org/debian|g' /etc/apt/sources.list && \ @@ -26,7 +29,7 @@ RUN cabal build && \ cp $(cabal list-bin shoresh-engine) /usr/local/bin/shoresh-engine # === RUNTIME STAGE === -FROM ubuntu:jammy-20240125 +FROM --platform=linux/amd64 ubuntu:jammy-20240125 RUN apt-get update && \ apt-get install -y --no-install-recommends \ diff --git a/deploy/docker/Dockerfile.talmudic_engine b/deploy/docker/Dockerfile.talmudic_engine index 5a5c95aa..5bf7df87 100644 --- a/deploy/docker/Dockerfile.talmudic_engine +++ b/deploy/docker/Dockerfile.talmudic_engine @@ -1,9 +1,12 @@ # Shorashim Talmudic Engine — SWI-Prolog dialectical reasoning # Construction spec §1.1: SWI-Prolog 9.2.2 # Communicates with vav_pipeline via Unix domain socket +# +# PLATFORM PIN: see Dockerfile.vav_pipeline for the rationale. +# Exoscale SKS = amd64; arm64 images cannot exec on the cluster. # === BUILD STAGE === -FROM swipl:9.2.2 AS build +FROM --platform=linux/amd64 swipl:9.2.2 AS build WORKDIR /app COPY talmudic_engine/ ./ @@ -12,7 +15,7 @@ COPY talmudic_engine/ ./ RUN swipl -g "consult('src/main'),halt(0)" -t "halt(1)" # === RUNTIME STAGE === -FROM swipl:9.2.2 +FROM --platform=linux/amd64 swipl:9.2.2 RUN useradd --create-home --shell /bin/bash shorashim USER shorashim diff --git a/deploy/docker/Dockerfile.vav_pipeline b/deploy/docker/Dockerfile.vav_pipeline index f4b917f1..db27ec6b 100644 --- a/deploy/docker/Dockerfile.vav_pipeline +++ b/deploy/docker/Dockerfile.vav_pipeline @@ -1,9 +1,18 @@ # Shorashim Vav Pipeline — Elixir/OTP release with Sanhedrin # Construction spec §1.1: Elixir 1.16.1, OTP 26.2.1 # The pipeline and Sanhedrin run in the same BEAM VM. +# +# PLATFORM PIN: +# Both FROM directives below pin `--platform=linux/amd64`. Exoscale +# SKS nodes are amd64; without this pin, a build on an Apple Silicon +# workstation would emit an arm64 image that the cluster cannot exec +# (the kohen's Day 3 fails with "exec format error"). The pin makes +# the build artifact self-correct regardless of build host. +# Local multi-arch dev (linux/amd64,linux/arm64) is supported by +# scripts/build_and_push.sh — see that script for the canonical build. # === BUILD STAGE === -FROM hexpm/elixir:1.16.1-erlang-26.2.1-ubuntu-jammy-20240125 AS build +FROM --platform=linux/amd64 hexpm/elixir:1.16.1-erlang-26.2.1-ubuntu-jammy-20240125 AS build RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/* @@ -28,7 +37,7 @@ RUN mix local.hex --force && \ RUN mix release # === RUNTIME STAGE === -FROM ubuntu:jammy-20240125 +FROM --platform=linux/amd64 ubuntu:jammy-20240125 RUN apt-get update && \ apt-get install -y --no-install-recommends \ diff --git a/deploy/k8s/deployment.yaml b/deploy/k8s/deployment.yaml index a05b5085..67b6724b 100644 --- a/deploy/k8s/deployment.yaml +++ b/deploy/k8s/deployment.yaml @@ -117,8 +117,17 @@ spec: cpu: "500m" # === KORBAN PROCESSOR (sidecar) === + # Korban_processor imports azazel_processor (the disposal + # channel) in the same Python process. The Python side writes + # AzazelRecord JSON files to /tmp/shorashim/azazel/pending_l3/; + # vav-pipeline's AzazelIngest GenServer watches that directory + # and ingests into L3. Both containers must mount the shared + # tmp-shorashim volume for the file bridge to work. - name: korban-processor image: exo.container-registry.com/the-recognition-protocol/korban-processor:latest + volumeMounts: + - name: tmp-shorashim + mountPath: /tmp resources: requests: memory: "128Mi" diff --git a/scripts/build_and_push.sh b/scripts/build_and_push.sh new file mode 100755 index 00000000..a5af4d9b --- /dev/null +++ b/scripts/build_and_push.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# +# build_and_push.sh — canonical builder procedure for shorashim images. +# +# Pin platform to linux/amd64 (Exoscale SKS) so a build on an Apple +# Silicon workstation does not silently ship arm64 binaries that the +# cluster cannot exec. The Dockerfiles also pin --platform=linux/amd64 +# on every FROM line as defense-in-depth; this script is the canonical +# build/push path that operators and CI both use. +# +# Default: linux/amd64 only (cluster-correct, fastest path). +# Optional: PLATFORMS=linux/amd64,linux/arm64 ./build_and_push.sh +# — multi-arch (slower; preserves local runnability for arm64 devs). +# +# Prerequisites: +# - Docker daemon running +# - docker buildx available +# - Logged in to exo.container-registry.com (CLI secret from +# Harbor → User Profile → CLI Secret) +# - Run from the repo root +# +# What gets pushed (four images): +# - vav-pipeline +# - shoresh-engine +# - talmudic-engine +# - korban-processor (azazel_processor is baked into this image) + +set -euo pipefail + +REGISTRY="${REGISTRY:-exo.container-registry.com}" +PROJECT="${PROJECT:-the-recognition-protocol}" +TAG="${TAG:-latest}" +PLATFORMS="${PLATFORMS:-linux/amd64}" + +# Sanity: must be run from repo root (where deploy/ and l1/ live) +if [[ ! -d "deploy/docker" || ! -d "l1/torah" ]]; then + echo "ERROR: run from the shorashim repo root (where deploy/ and l1/ live)" >&2 + exit 1 +fi + +# Sanity: docker buildx +if ! docker buildx version >/dev/null 2>&1; then + echo "ERROR: docker buildx not available — install Docker Desktop or buildx plugin" >&2 + exit 1 +fi + +# Sanity: registry login. We probe with a HEAD against the registry +# v2 endpoint; if 401, login is missing. +if ! docker info 2>/dev/null | grep -q "$REGISTRY"; then + echo "Reminder: run 'docker login $REGISTRY' first (CLI secret from Harbor User Profile)." + echo "Continuing — push will fail clearly if not logged in." +fi + +# Create or use the shorashim buildx builder +if ! docker buildx inspect shorashim-builder >/dev/null 2>&1; then + echo ">> creating buildx builder 'shorashim-builder'" + docker buildx create --name shorashim-builder --use +else + docker buildx use shorashim-builder +fi +docker buildx inspect --bootstrap >/dev/null + +# Source dir → image name → Dockerfile mapping. +# korban_processor's Dockerfile also COPYs azazel_processor (see +# Dockerfile.korban_processor — AZAZEL CO-RESIDENT section). +build_one() { + local component="$1" + local image="$2" + local dockerfile="deploy/docker/Dockerfile.${component}" + + if [[ ! -f "$dockerfile" ]]; then + echo "ERROR: $dockerfile not found" >&2 + return 1 + fi + + echo "" + echo "================================================================" + echo ">> building ${image}:${TAG} for ${PLATFORMS}" + echo " dockerfile: ${dockerfile}" + echo "================================================================" + + docker buildx build \ + --platform "$PLATFORMS" \ + --file "$dockerfile" \ + --tag "${REGISTRY}/${PROJECT}/${image}:${TAG}" \ + --push \ + . +} + +build_one vav_pipeline vav-pipeline +build_one shoresh_engine shoresh-engine +build_one talmudic_engine talmudic-engine +build_one korban_processor korban-processor + +echo "" +echo "================================================================" +echo ">> all four images built and pushed for ${PLATFORMS}" +echo "================================================================" +echo "" +echo "Verify from the cluster (after kohen has kubeconfig loaded):" +echo " kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}'" +echo " # expected: amd64" +echo "" +echo "Verify image manifests:" +echo " for img in vav-pipeline shoresh-engine talmudic-engine korban-processor; do" +echo " docker manifest inspect \"${REGISTRY}/${PROJECT}/\$img:${TAG}\" \\" +echo " | jq -r '.architecture // .manifests[].platform.architecture' \\" +echo " | sort -u" +echo " done" +echo "" +echo "Then re-run kohen gate 0i in KOHEN_PROMPT.md."