deployment: split the migration role from the application role (kohen Day-3 42501)

TORAH ANCHOR — Vayikra 18:5-6 / the gilui arayot shape:
The kohen refused the easy fix on the grounds that dissolving a
required distinction for convenience is the boundary violation this
system's own halt rule names. That reading is correct and this commit
implements it rather than working around it.

THE BLOCKER (kohen Day 3, escalated):

  ** (Postgrex.Error) ERROR 42501 (insufficient_privilege)
     permission denied for schema public
     Could not create schema migrations table.

Ecto.Migrator.run/3 calls ensure_schema_migrations_table! BEFORE it
looks at what is pending (see vav_pipeline/lib/vav_pipeline/
release.ex — migrate/0 wraps Ecto.Migrator.run(&1, :up, all: true)).
So the migrate init container issues

  CREATE TABLE IF NOT EXISTS "schema_migrations"

on EVERY pod start, even now that 20260428000001_create_azazel_records
is applied and nothing is pending. As shorashim_app — which has USAGE
but not CREATE on schema public, correctly — that is a permanent
failure and the pod never reaches Running.

ROOT CAUSE: the manifest did not implement what the runbook already
specified. KOHEN.md:184 says "the migration runs as avnadmin; the
application runs as shorashim_app", and then deployment.yaml gave the
init container and the app containers the same shorashim-secrets.
The two-role split existed as prose and not as configuration.

THE FIX:

  deploy/k8s/deployment.yaml — migrate init container only:
    + env:
    +   - name: DATABASE_URL
    +     valueFrom:
    +       secretKeyRef:
    +         name: shorashim-migrate-secrets
    +         key: DATABASE_URL

An explicit `env:` entry rather than swapping the secretRef, as the
kohen proposed. Same effect today, and strictly safer tomorrow:
`env:` takes precedence over every `envFrom` source, so exactly ONE
variable changes role and anything else that later lands in
shorashim-secrets is still inherited by the migrator. Swapping the
ref would have silently dropped such a key from the init container.

App containers keep shorashim-secrets untouched. Append-only holds:
the L3 tables stay avnadmin-owned with the REVOKEs in place, and
shorashim_app still has only INSERT/SELECT.

WHY NOT GRANT shorashim_app CREATE ON SCHEMA public — recorded, not
just decided. The kohen considered it, noted append-only would
technically survive, and rejected it because it hands the application
DDL power over the schema it writes into and collapses the app/
migration boundary KOHEN.md draws on purpose. Concurred, and the
reasoning is now written into deployment.yaml and secrets.yaml at the
two places someone would next hit 42501 — so the shortcut cannot be
quietly re-opened by whoever meets the error without this thread.

ALSO IN THIS COMMIT:

deploy/k8s/secrets.yaml — shorashim-migrate-secrets added as a
TEMPLATE (placeholder value, no credential), with the create command.
The command differs from the one in the escalation in two ways, both
deliberate:

  1. The userinfo rewrite is done in Python, not with
     `sed "s|//shorashim_app:[^@]*@|//avnadmin:${AVN_PW}@|"`. A
     generated password containing `&` re-inserts the matched text,
     one containing `|` ends the sed expression early, and `\` escapes.
     The failure is silent — the result is a DATABASE_URL that looks
     plausible and does not authenticate, which on Day 3 would read as
     a second, different blocker. urllib.parse.quote(safe="") also
     percent-encodes anything not URL-safe in the userinfo field,
     which sed passes through raw.
  2. The value goes through a `umask 077` temp file rather than
     --from-literal, so the avnadmin password is never an argv entry
     visible to `ps` on a shared host.

KOHEN.md — new step 3a-pre (create/verify the secret before applying),
and §Day-3b now says configmap.yaml is not optional on this pass
(it carries SHORESH_ENGINE_BINARY) and that secrets.yaml must NOT be
applied — it is a template and would overwrite both live secrets with
placeholders.

DELIBERATELY NO FALLBACK: if shorashim-migrate-secrets is absent the
pod stops at CreateContainerConfigError. Loud is correct here. A
fallback to the application role would reintroduce 42501 at a moment
when everything else looked healthy.

NOT DONE BY THE BUILDER: the secret itself. It carries a live DDL
credential and it is cluster state; both are the kohen's hands, not
mine. The command is in deploy/k8s/secrets.yaml.

VERIFIED:
- kubectl apply --dry-run=client on deployment.yaml, configmap.yaml
  and secrets.yaml — all four objects valid (client-side only; no
  cluster state touched)
- env/envFrom wiring re-read from the parsed manifest: migrate has
  envFrom [shorashim-config, shorashim-secrets] + env DATABASE_URL
  from shorashim-migrate-secrets; vav-pipeline has envFrom
  [shorashim-config, shorashim-secrets] and no env override
- Doorpost.verify_identity() :ok — no @spec_files entry touched, so
  no @expected_hash change
- l1/integrity.sha256 unchanged: 99bbd07a…4ea377

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KOHEN.md

@@ -186,6 +186,10 @@ REVOKE DELETE ON teiku_registry FROM shorashim_app;
 
 If you create `shorashim_app`, **rotate** `DATABASE_URL` to use it instead of `avnadmin` (the migration runs as `avnadmin`; the application runs as `shorashim_app`).
 
+**Two roles means two secrets.** That sentence is a manifest requirement, not just advice. `shorashim-secrets` carries the `shorashim_app` URL and is read by every app container; `shorashim-migrate-secrets` carries the `avnadmin` URL and is read ONLY by the `migrate` init container, which overrides `DATABASE_URL` for itself alone. Create the second secret at step **3a-pre** below before applying the deployment — `deploy/k8s/secrets.yaml` carries the exact command.
+
+Do **not** close the gap by granting `shorashim_app` `CREATE ON SCHEMA public` instead. `Ecto.Migrator.run/3` calls `ensure_schema_migrations_table!` before it checks what is pending, so it issues `CREATE TABLE IF NOT EXISTS schema_migrations` on every pod start even when migrations are current; as `shorashim_app` that is `ERROR 42501 permission denied for schema public` and the pod never reaches `Running`. Granting DDL to the application would clear the error and dissolve the distinction this line draws — the *gilui arayot* shape the system's own halt rule names. Two secrets is the one-line answer.
+
 ### Day 3 — *Meshichah* (Vayikra 8:10) — Apply the deployment
 
 Anoint the mishkan. Bring the system into running state.
@@ -204,6 +208,16 @@ L1 ships with the vessel. The scroll is sealed in the aron. The build artifact c
 #   --from-file=l1/torah
 ```
 
+#### 3a-pre. Create the migration-role secret (once)
+
+The `migrate` init container needs `avnadmin`, not `shorashim_app`. Without this secret the pod stops at `CreateContainerConfigError` — deliberately, because there is no fallback to the application role.
+
+```bash
+kubectl get secret shorashim-migrate-secrets -n shorashim
+```
+
+If it is absent, create it with the command in `deploy/k8s/secrets.yaml` (it reads the avnadmin password from `exo dbaas user reveal-secrets`, rewrites the userinfo of the existing app URL in Python, and passes the value through a `umask 077` temp file rather than `--from-literal`). Do not hand-edit the URL with `sed`: a generated password containing `&`, `|`, or `\` corrupts a `sed` replacement silently and yields a URL that looks right and does not authenticate.
+
 #### 3b. Apply the manifests
 
 ```bash
@@ -213,6 +227,10 @@ kubectl apply -f deploy/k8s/deployment.yaml
 kubectl apply -f deploy/k8s/service.yaml
 ```
 
+`configmap.yaml` is not optional on this pass — it now carries `SHORESH_ENGINE_BINARY`, which the Erlang Port needs.
+
+**Do not** `kubectl apply -f deploy/k8s/secrets.yaml`. It is a template with placeholder values and would overwrite both live secrets.
+
 Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `emptyDir` (lost on pod restart — kohen re-writes today's provision per the daily-ops section). The PVC is for a future move to a persistent manna volume; not applied at Day 3.
 
 #### 3c. Watch the pod come up

deploy/k8s/deployment.yaml

@@ -130,6 +130,46 @@ spec:
               mountPath: /opt/shoresh
 
         # Run L3 migrations
+        #
+        # TWO ROLES, ONE IMAGE. KOHEN.md:184 — "the migration runs as
+        # avnadmin; the application runs as shorashim_app." This init
+        # container was inheriting the APPLICATION's DATABASE_URL and so
+        # ran DDL as shorashim_app, which does not have CREATE on schema
+        # public and never should:
+        #   ** (Postgrex.Error) ERROR 42501 (insufficient_privilege)
+        #      permission denied for schema public
+        #      Could not create schema migrations table.
+        #
+        # Ecto.Migrator.run/3 calls ensure_schema_migrations_table!
+        # BEFORE it looks at what is pending, so this fired on every pod
+        # start even with migrations already current — the pod could
+        # never reach Running. See vav_pipeline/lib/vav_pipeline/
+        # release.ex.
+        #
+        # WHY NOT JUST GRANT shorashim_app CREATE ON SCHEMA public:
+        # because that is the boundary, not an obstacle in front of it.
+        # Append-only would technically survive (the L3 tables are
+        # avnadmin-owned with REVOKEs), but the application would hold
+        # DDL power over the schema it writes into, and the app/migration
+        # distinction KOHEN.md draws on purpose would exist only as a
+        # convention. Vayikra 18 — a required distinction dissolved for
+        # convenience is the gilui arayot shape, and the halt rule this
+        # system runs on names exactly that. The kohen refused it on
+        # those grounds and was right to; recording the reasoning here so
+        # the next person to hit 42501 does not quietly re-open it.
+        #
+        # DATABASE_URL is overridden as an explicit `env:` entry rather
+        # than by swapping the secretRef. `env:` takes precedence over
+        # every `envFrom` source, so exactly ONE variable changes role
+        # and anything else that lands in shorashim-secrets later is
+        # still inherited by the migrator.
+        #
+        # shorashim-migrate-secrets is NOT in this repo and must not be.
+        # It carries the DDL-capable credential; see
+        # deploy/k8s/secrets.yaml for the shape and the create command.
+        # If it is absent the pod fails at CreateContainerConfigError —
+        # loudly, which is correct. There is deliberately no fallback to
+        # the application role.
         - name: migrate
           image: exo.container-registry.com/the-recognition-protocol/vav-pipeline:latest
           command: ["./app/bin/vav_pipeline", "eval", "VavPipeline.Release.migrate()"]
@@ -138,6 +178,14 @@ spec:
                 name: shorashim-config
             - secretRef:
                 name: shorashim-secrets
+          env:
+            # DDL role — avnadmin, not shorashim_app. Overrides the
+            # DATABASE_URL that shorashim-secrets supplies above.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: shorashim-migrate-secrets
+                  key: DATABASE_URL
 
       # --- CONTAINERS ---
       containers:

deploy/k8s/secrets.yaml

@@ -12,6 +12,79 @@ metadata:
   namespace: shorashim
 type: Opaque
 stringData:
-  # PostgreSQL connection for L3 Oral Tradition
+  # PostgreSQL connection for L3 Oral Tradition — the APPLICATION role.
   # Format: ecto://user:password@host:port/database?sslmode=require
+  # This is shorashim_app: INSERT and SELECT on the L3 tables, no DDL,
+  # no CREATE on schema public. Append-only is enforced by the grants,
+  # not by the application's good behaviour.
   DATABASE_URL: "REPLACE_WITH_ACTUAL_DATABASE_URL"
+
+---
+# ============================================================
+# shorashim-migrate-secrets — the DDL role
+# ============================================================
+#
+# KOHEN.md:184 — "the migration runs as avnadmin; the application runs
+# as shorashim_app." TWO ROLES, TWO SECRETS. Only the `migrate` init
+# container reads this one (deployment.yaml overrides DATABASE_URL for
+# that container alone); every app container keeps shorashim-secrets
+# above.
+#
+# WHY IT IS SEPARATE AND NOT JUST A GRANT: Ecto.Migrator.run/3 calls
+# ensure_schema_migrations_table! BEFORE checking what is pending, so
+# it issues CREATE TABLE IF NOT EXISTS schema_migrations on EVERY pod
+# start, even with migrations current. As shorashim_app that is
+# ERROR 42501 permission denied for schema public, and the pod never
+# reaches Running. The alternative — granting shorashim_app CREATE on
+# schema public — collapses the app/migration boundary the runbook
+# draws deliberately. Do not take it.
+#
+# THIS FILE IS A TEMPLATE. Never commit a real value. Create the secret
+# from the live database credential instead:
+#
+#   export KUBECONFIG=~/.kube/exoscale-temple-council.yaml
+#   umask 077
+#   TMP=$(mktemp)
+#
+#   exo dbaas user reveal-secrets temple-council-db avnadmin \
+#       --zone ch-gva-2 -O json \
+#   | python3 -c '
+#   import base64, json, subprocess, sys, urllib.parse
+#   avn = json.load(sys.stdin)["password"]
+#   app = subprocess.run(
+#       ["kubectl", "get", "secret", "shorashim-secrets", "-n", "shorashim",
+#        "-o", "jsonpath={.data.DATABASE_URL}"],
+#       capture_output=True, text=True, check=True).stdout
+#   u = urllib.parse.urlsplit(base64.b64decode(app).decode())
+#   pw = urllib.parse.quote(avn, safe="")
+#   netloc = "avnadmin:%s@%s:%d" % (pw, u.hostname, u.port)
+#   sys.stdout.write(
+#       urllib.parse.urlunsplit((u.scheme, netloc, u.path, u.query, u.fragment)))
+#   ' > "$TMP"
+#
+#   kubectl create secret generic shorashim-migrate-secrets \
+#     --namespace shorashim --from-file=DATABASE_URL="$TMP"
+#
+#   rm -f "$TMP"
+#
+# The rewrite is done in Python, not with `sed s|...|...|`, on purpose.
+# A generated password containing & or | or \ corrupts a sed
+# replacement silently — & re-inserts the matched text and | ends the
+# expression early — and the result is a DATABASE_URL that looks
+# plausible and does not authenticate. quote(safe="") also percent-
+# encodes any character that is not URL-safe in the userinfo field,
+# which sed would have passed through raw. The value goes via a
+# umask-077 temp file rather than --from-literal so the password is
+# never an argv entry visible to `ps`.
+#
+# Absence is meant to be loud: without this secret the pod stops at
+# CreateContainerConfigError. There is no fallback to the app role.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: shorashim-migrate-secrets
+  namespace: shorashim
+type: Opaque
+stringData:
+  # ecto://avnadmin:PASSWORD@host:port/database?sslmode=require
+  DATABASE_URL: "REPLACE_WITH_AVNADMIN_DATABASE_URL"