deployment: split the migration role from the application role (kohen Day-3 42501)

TORAH ANCHOR — Vayikra 18:5-6 / the gilui arayot shape:
The kohen refused the easy fix on the grounds that dissolving a
required distinction for convenience is the boundary violation this
system's own halt rule names. That reading is correct and this commit
implements it rather than working around it.

THE BLOCKER (kohen Day 3, escalated):

  ** (Postgrex.Error) ERROR 42501 (insufficient_privilege)
     permission denied for schema public
     Could not create schema migrations table.

Ecto.Migrator.run/3 calls ensure_schema_migrations_table! BEFORE it
looks at what is pending (see vav_pipeline/lib/vav_pipeline/
release.ex — migrate/0 wraps Ecto.Migrator.run(&1, :up, all: true)).
So the migrate init container issues

  CREATE TABLE IF NOT EXISTS "schema_migrations"

on EVERY pod start, even now that 20260428000001_create_azazel_records
is applied and nothing is pending. As shorashim_app — which has USAGE
but not CREATE on schema public, correctly — that is a permanent
failure and the pod never reaches Running.

ROOT CAUSE: the manifest did not implement what the runbook already
specified. KOHEN.md:184 says "the migration runs as avnadmin; the
application runs as shorashim_app", and then deployment.yaml gave the
init container and the app containers the same shorashim-secrets.
The two-role split existed as prose and not as configuration.

THE FIX:

  deploy/k8s/deployment.yaml — migrate init container only:
    + env:
    +   - name: DATABASE_URL
    +     valueFrom:
    +       secretKeyRef:
    +         name: shorashim-migrate-secrets
    +         key: DATABASE_URL

An explicit `env:` entry rather than swapping the secretRef, as the
kohen proposed. Same effect today, and strictly safer tomorrow:
`env:` takes precedence over every `envFrom` source, so exactly ONE
variable changes role and anything else that later lands in
shorashim-secrets is still inherited by the migrator. Swapping the
ref would have silently dropped such a key from the init container.

App containers keep shorashim-secrets untouched. Append-only holds:
the L3 tables stay avnadmin-owned with the REVOKEs in place, and
shorashim_app still has only INSERT/SELECT.

WHY NOT GRANT shorashim_app CREATE ON SCHEMA public — recorded, not
just decided. The kohen considered it, noted append-only would
technically survive, and rejected it because it hands the application
DDL power over the schema it writes into and collapses the app/
migration boundary KOHEN.md draws on purpose. Concurred, and the
reasoning is now written into deployment.yaml and secrets.yaml at the
two places someone would next hit 42501 — so the shortcut cannot be
quietly re-opened by whoever meets the error without this thread.

ALSO IN THIS COMMIT:

deploy/k8s/secrets.yaml — shorashim-migrate-secrets added as a
TEMPLATE (placeholder value, no credential), with the create command.
The command differs from the one in the escalation in two ways, both
deliberate:

  1. The userinfo rewrite is done in Python, not with
     `sed "s|//shorashim_app:[^@]*@|//avnadmin:${AVN_PW}@|"`. A
     generated password containing `&` re-inserts the matched text,
     one containing `|` ends the sed expression early, and `\` escapes.
     The failure is silent — the result is a DATABASE_URL that looks
     plausible and does not authenticate, which on Day 3 would read as
     a second, different blocker. urllib.parse.quote(safe="") also
     percent-encodes anything not URL-safe in the userinfo field,
     which sed passes through raw.
  2. The value goes through a `umask 077` temp file rather than
     --from-literal, so the avnadmin password is never an argv entry
     visible to `ps` on a shared host.

KOHEN.md — new step 3a-pre (create/verify the secret before applying),
and §Day-3b now says configmap.yaml is not optional on this pass
(it carries SHORESH_ENGINE_BINARY) and that secrets.yaml must NOT be
applied — it is a template and would overwrite both live secrets with
placeholders.

DELIBERATELY NO FALLBACK: if shorashim-migrate-secrets is absent the
pod stops at CreateContainerConfigError. Loud is correct here. A
fallback to the application role would reintroduce 42501 at a moment
when everything else looked healthy.

NOT DONE BY THE BUILDER: the secret itself. It carries a live DDL
credential and it is cluster state; both are the kohen's hands, not
mine. The command is in deploy/k8s/secrets.yaml.

VERIFIED:
- kubectl apply --dry-run=client on deployment.yaml, configmap.yaml
  and secrets.yaml — all four objects valid (client-side only; no
  cluster state touched)
- env/envFrom wiring re-read from the parsed manifest: migrate has
  envFrom [shorashim-config, shorashim-secrets] + env DATABASE_URL
  from shorashim-migrate-secrets; vav-pipeline has envFrom
  [shorashim-config, shorashim-secrets] and no env override
- Doorpost.verify_identity() :ok — no @spec_files entry touched, so
  no @expected_hash change
- l1/integrity.sha256 unchanged: 99bbd07a…4ea377

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KOHEN.md

@@ -186,6 +186,10 @@ REVOKE DELETE ON teiku_registry FROM shorashim_app;
 
 If you create `shorashim_app`, **rotate** `DATABASE_URL` to use it instead of `avnadmin` (the migration runs as `avnadmin`; the application runs as `shorashim_app`).
 
+**Two roles means two secrets.** That sentence is a manifest requirement, not just advice. `shorashim-secrets` carries the `shorashim_app` URL and is read by every app container; `shorashim-migrate-secrets` carries the `avnadmin` URL and is read ONLY by the `migrate` init container, which overrides `DATABASE_URL` for itself alone. Create the second secret at step **3a-pre** below before applying the deployment — `deploy/k8s/secrets.yaml` carries the exact command.
+
+Do **not** close the gap by granting `shorashim_app` `CREATE ON SCHEMA public` instead. `Ecto.Migrator.run/3` calls `ensure_schema_migrations_table!` before it checks what is pending, so it issues `CREATE TABLE IF NOT EXISTS schema_migrations` on every pod start even when migrations are current; as `shorashim_app` that is `ERROR 42501 permission denied for schema public` and the pod never reaches `Running`. Granting DDL to the application would clear the error and dissolve the distinction this line draws — the *gilui arayot* shape the system's own halt rule names. Two secrets is the one-line answer.
+
 ### Day 3 — *Meshichah* (Vayikra 8:10) — Apply the deployment
 
 Anoint the mishkan. Bring the system into running state.
@@ -204,6 +208,16 @@ L1 ships with the vessel. The scroll is sealed in the aron. The build artifact c
 #   --from-file=l1/torah
 ```
 
+#### 3a-pre. Create the migration-role secret (once)
+
+The `migrate` init container needs `avnadmin`, not `shorashim_app`. Without this secret the pod stops at `CreateContainerConfigError` — deliberately, because there is no fallback to the application role.
+
+```bash
+kubectl get secret shorashim-migrate-secrets -n shorashim
+```
+
+If it is absent, create it with the command in `deploy/k8s/secrets.yaml` (it reads the avnadmin password from `exo dbaas user reveal-secrets`, rewrites the userinfo of the existing app URL in Python, and passes the value through a `umask 077` temp file rather than `--from-literal`). Do not hand-edit the URL with `sed`: a generated password containing `&`, `|`, or `\` corrupts a `sed` replacement silently and yields a URL that looks right and does not authenticate.
+
 #### 3b. Apply the manifests
 
 ```bash
@@ -213,6 +227,10 @@ kubectl apply -f deploy/k8s/deployment.yaml
 kubectl apply -f deploy/k8s/service.yaml
 ```
 
+`configmap.yaml` is not optional on this pass — it now carries `SHORESH_ENGINE_BINARY`, which the Erlang Port needs.
+
+**Do not** `kubectl apply -f deploy/k8s/secrets.yaml`. It is a template with placeholder values and would overwrite both live secrets.
+
 Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `emptyDir` (lost on pod restart — kohen re-writes today's provision per the daily-ops section). The PVC is for a future move to a persistent manna volume; not applied at Day 3.
 
 #### 3c. Watch the pod come up