diff --git a/KOHEN.md b/KOHEN.md index b6e6382a..d01119f4 100644 --- a/KOHEN.md +++ b/KOHEN.md @@ -186,6 +186,10 @@ REVOKE DELETE ON teiku_registry FROM shorashim_app; If you create `shorashim_app`, **rotate** `DATABASE_URL` to use it instead of `avnadmin` (the migration runs as `avnadmin`; the application runs as `shorashim_app`). +**Two roles means two secrets.** That sentence is a manifest requirement, not just advice. `shorashim-secrets` carries the `shorashim_app` URL and is read by every app container; `shorashim-migrate-secrets` carries the `avnadmin` URL and is read ONLY by the `migrate` init container, which overrides `DATABASE_URL` for itself alone. Create the second secret at step **3a-pre** below before applying the deployment — `deploy/k8s/secrets.yaml` carries the exact command. + +Do **not** close the gap by granting `shorashim_app` `CREATE ON SCHEMA public` instead. `Ecto.Migrator.run/3` calls `ensure_schema_migrations_table!` before it checks what is pending, so it issues `CREATE TABLE IF NOT EXISTS schema_migrations` on every pod start even when migrations are current; as `shorashim_app` that is `ERROR 42501 permission denied for schema public` and the pod never reaches `Running`. Granting DDL to the application would clear the error and dissolve the distinction this line draws — the *gilui arayot* shape the system's own halt rule names. Two secrets is the one-line answer. + ### Day 3 — *Meshichah* (Vayikra 8:10) — Apply the deployment Anoint the mishkan. Bring the system into running state. @@ -204,6 +208,16 @@ L1 ships with the vessel. The scroll is sealed in the aron. The build artifact c # --from-file=l1/torah ``` +#### 3a-pre. Create the migration-role secret (once) + +The `migrate` init container needs `avnadmin`, not `shorashim_app`. Without this secret the pod stops at `CreateContainerConfigError` — deliberately, because there is no fallback to the application role. + +```bash +kubectl get secret shorashim-migrate-secrets -n shorashim +``` + +If it is absent, create it with the command in `deploy/k8s/secrets.yaml` (it reads the avnadmin password from `exo dbaas user reveal-secrets`, rewrites the userinfo of the existing app URL in Python, and passes the value through a `umask 077` temp file rather than `--from-literal`). Do not hand-edit the URL with `sed`: a generated password containing `&`, `|`, or `\` corrupts a `sed` replacement silently and yields a URL that looks right and does not authenticate. + #### 3b. Apply the manifests ```bash @@ -213,6 +227,10 @@ kubectl apply -f deploy/k8s/deployment.yaml kubectl apply -f deploy/k8s/service.yaml ``` +`configmap.yaml` is not optional on this pass — it now carries `SHORESH_ENGINE_BINARY`, which the Erlang Port needs. + +**Do not** `kubectl apply -f deploy/k8s/secrets.yaml`. It is a template with placeholder values and would overwrite both live secrets. + Note: `manna-volume.yaml` defines a PVC but `deployment.yaml` mounts manna as `emptyDir` (lost on pod restart — kohen re-writes today's provision per the daily-ops section). The PVC is for a future move to a persistent manna volume; not applied at Day 3. #### 3c. Watch the pod come up diff --git a/deploy/k8s/deployment.yaml b/deploy/k8s/deployment.yaml index 1e473469..99ec9ba4 100644 --- a/deploy/k8s/deployment.yaml +++ b/deploy/k8s/deployment.yaml @@ -130,6 +130,46 @@ spec: mountPath: /opt/shoresh # Run L3 migrations + # + # TWO ROLES, ONE IMAGE. KOHEN.md:184 — "the migration runs as + # avnadmin; the application runs as shorashim_app." This init + # container was inheriting the APPLICATION's DATABASE_URL and so + # ran DDL as shorashim_app, which does not have CREATE on schema + # public and never should: + # ** (Postgrex.Error) ERROR 42501 (insufficient_privilege) + # permission denied for schema public + # Could not create schema migrations table. + # + # Ecto.Migrator.run/3 calls ensure_schema_migrations_table! + # BEFORE it looks at what is pending, so this fired on every pod + # start even with migrations already current — the pod could + # never reach Running. See vav_pipeline/lib/vav_pipeline/ + # release.ex. + # + # WHY NOT JUST GRANT shorashim_app CREATE ON SCHEMA public: + # because that is the boundary, not an obstacle in front of it. + # Append-only would technically survive (the L3 tables are + # avnadmin-owned with REVOKEs), but the application would hold + # DDL power over the schema it writes into, and the app/migration + # distinction KOHEN.md draws on purpose would exist only as a + # convention. Vayikra 18 — a required distinction dissolved for + # convenience is the gilui arayot shape, and the halt rule this + # system runs on names exactly that. The kohen refused it on + # those grounds and was right to; recording the reasoning here so + # the next person to hit 42501 does not quietly re-open it. + # + # DATABASE_URL is overridden as an explicit `env:` entry rather + # than by swapping the secretRef. `env:` takes precedence over + # every `envFrom` source, so exactly ONE variable changes role + # and anything else that lands in shorashim-secrets later is + # still inherited by the migrator. + # + # shorashim-migrate-secrets is NOT in this repo and must not be. + # It carries the DDL-capable credential; see + # deploy/k8s/secrets.yaml for the shape and the create command. + # If it is absent the pod fails at CreateContainerConfigError — + # loudly, which is correct. There is deliberately no fallback to + # the application role. - name: migrate image: exo.container-registry.com/the-recognition-protocol/vav-pipeline:latest command: ["./app/bin/vav_pipeline", "eval", "VavPipeline.Release.migrate()"] @@ -138,6 +178,14 @@ spec: name: shorashim-config - secretRef: name: shorashim-secrets + env: + # DDL role — avnadmin, not shorashim_app. Overrides the + # DATABASE_URL that shorashim-secrets supplies above. + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: shorashim-migrate-secrets + key: DATABASE_URL # --- CONTAINERS --- containers: diff --git a/deploy/k8s/secrets.yaml b/deploy/k8s/secrets.yaml index e95ed2ca..3b3fdaaa 100644 --- a/deploy/k8s/secrets.yaml +++ b/deploy/k8s/secrets.yaml @@ -12,6 +12,79 @@ metadata: namespace: shorashim type: Opaque stringData: - # PostgreSQL connection for L3 Oral Tradition + # PostgreSQL connection for L3 Oral Tradition — the APPLICATION role. # Format: ecto://user:password@host:port/database?sslmode=require + # This is shorashim_app: INSERT and SELECT on the L3 tables, no DDL, + # no CREATE on schema public. Append-only is enforced by the grants, + # not by the application's good behaviour. DATABASE_URL: "REPLACE_WITH_ACTUAL_DATABASE_URL" + +--- +# ============================================================ +# shorashim-migrate-secrets — the DDL role +# ============================================================ +# +# KOHEN.md:184 — "the migration runs as avnadmin; the application runs +# as shorashim_app." TWO ROLES, TWO SECRETS. Only the `migrate` init +# container reads this one (deployment.yaml overrides DATABASE_URL for +# that container alone); every app container keeps shorashim-secrets +# above. +# +# WHY IT IS SEPARATE AND NOT JUST A GRANT: Ecto.Migrator.run/3 calls +# ensure_schema_migrations_table! BEFORE checking what is pending, so +# it issues CREATE TABLE IF NOT EXISTS schema_migrations on EVERY pod +# start, even with migrations current. As shorashim_app that is +# ERROR 42501 permission denied for schema public, and the pod never +# reaches Running. The alternative — granting shorashim_app CREATE on +# schema public — collapses the app/migration boundary the runbook +# draws deliberately. Do not take it. +# +# THIS FILE IS A TEMPLATE. Never commit a real value. Create the secret +# from the live database credential instead: +# +# export KUBECONFIG=~/.kube/exoscale-temple-council.yaml +# umask 077 +# TMP=$(mktemp) +# +# exo dbaas user reveal-secrets temple-council-db avnadmin \ +# --zone ch-gva-2 -O json \ +# | python3 -c ' +# import base64, json, subprocess, sys, urllib.parse +# avn = json.load(sys.stdin)["password"] +# app = subprocess.run( +# ["kubectl", "get", "secret", "shorashim-secrets", "-n", "shorashim", +# "-o", "jsonpath={.data.DATABASE_URL}"], +# capture_output=True, text=True, check=True).stdout +# u = urllib.parse.urlsplit(base64.b64decode(app).decode()) +# pw = urllib.parse.quote(avn, safe="") +# netloc = "avnadmin:%s@%s:%d" % (pw, u.hostname, u.port) +# sys.stdout.write( +# urllib.parse.urlunsplit((u.scheme, netloc, u.path, u.query, u.fragment))) +# ' > "$TMP" +# +# kubectl create secret generic shorashim-migrate-secrets \ +# --namespace shorashim --from-file=DATABASE_URL="$TMP" +# +# rm -f "$TMP" +# +# The rewrite is done in Python, not with `sed s|...|...|`, on purpose. +# A generated password containing & or | or \ corrupts a sed +# replacement silently — & re-inserts the matched text and | ends the +# expression early — and the result is a DATABASE_URL that looks +# plausible and does not authenticate. quote(safe="") also percent- +# encodes any character that is not URL-safe in the userinfo field, +# which sed would have passed through raw. The value goes via a +# umask-077 temp file rather than --from-literal so the password is +# never an argv entry visible to `ps`. +# +# Absence is meant to be loud: without this secret the pod stops at +# CreateContainerConfigError. There is no fallback to the app role. +apiVersion: v1 +kind: Secret +metadata: + name: shorashim-migrate-secrets + namespace: shorashim +type: Opaque +stringData: + # ecto://avnadmin:PASSWORD@host:port/database?sslmode=require + DATABASE_URL: "REPLACE_WITH_AVNADMIN_DATABASE_URL"