Scott KarpandClaude Opus 5
6e34001c58
TORAH ANCHOR — Vayikra 18:5-6 / the gilui arayot shape:
The kohen refused the easy fix on the grounds that dissolving a
required distinction for convenience is the boundary violation this
system's own halt rule names. That reading is correct and this commit
implements it rather than working around it.
THE BLOCKER (kohen Day 3, escalated):
** (Postgrex.Error) ERROR 42501 (insufficient_privilege)
permission denied for schema public
Could not create schema migrations table.
Ecto.Migrator.run/3 calls ensure_schema_migrations_table! BEFORE it
looks at what is pending (see vav_pipeline/lib/vav_pipeline/
release.ex — migrate/0 wraps Ecto.Migrator.run(&1, :up, all: true)).
So the migrate init container issues
CREATE TABLE IF NOT EXISTS "schema_migrations"
on EVERY pod start, even now that 20260428000001_create_azazel_records
is applied and nothing is pending. As shorashim_app — which has USAGE
but not CREATE on schema public, correctly — that is a permanent
failure and the pod never reaches Running.
ROOT CAUSE: the manifest did not implement what the runbook already
specified. KOHEN.md:184 says "the migration runs as avnadmin; the
application runs as shorashim_app", and then deployment.yaml gave the
init container and the app containers the same shorashim-secrets.
The two-role split existed as prose and not as configuration.
THE FIX:
deploy/k8s/deployment.yaml — migrate init container only:
+ env:
+ - name: DATABASE_URL
+ valueFrom:
+ secretKeyRef:
+ name: shorashim-migrate-secrets
+ key: DATABASE_URL
An explicit `env:` entry rather than swapping the secretRef, as the
kohen proposed. Same effect today, and strictly safer tomorrow:
`env:` takes precedence over every `envFrom` source, so exactly ONE
variable changes role and anything else that later lands in
shorashim-secrets is still inherited by the migrator. Swapping the
ref would have silently dropped such a key from the init container.
App containers keep shorashim-secrets untouched. Append-only holds:
the L3 tables stay avnadmin-owned with the REVOKEs in place, and
shorashim_app still has only INSERT/SELECT.
WHY NOT GRANT shorashim_app CREATE ON SCHEMA public — recorded, not
just decided. The kohen considered it, noted append-only would
technically survive, and rejected it because it hands the application
DDL power over the schema it writes into and collapses the app/
migration boundary KOHEN.md draws on purpose. Concurred, and the
reasoning is now written into deployment.yaml and secrets.yaml at the
two places someone would next hit 42501 — so the shortcut cannot be
quietly re-opened by whoever meets the error without this thread.
ALSO IN THIS COMMIT:
deploy/k8s/secrets.yaml — shorashim-migrate-secrets added as a
TEMPLATE (placeholder value, no credential), with the create command.
The command differs from the one in the escalation in two ways, both
deliberate:
1. The userinfo rewrite is done in Python, not with
`sed "s|//shorashim_app:[^@]*@|//avnadmin:${AVN_PW}@|"`. A
generated password containing `&` re-inserts the matched text,
one containing `|` ends the sed expression early, and `\` escapes.
The failure is silent — the result is a DATABASE_URL that looks
plausible and does not authenticate, which on Day 3 would read as
a second, different blocker. urllib.parse.quote(safe="") also
percent-encodes anything not URL-safe in the userinfo field,
which sed passes through raw.
2. The value goes through a `umask 077` temp file rather than
--from-literal, so the avnadmin password is never an argv entry
visible to `ps` on a shared host.
KOHEN.md — new step 3a-pre (create/verify the secret before applying),
and §Day-3b now says configmap.yaml is not optional on this pass
(it carries SHORESH_ENGINE_BINARY) and that secrets.yaml must NOT be
applied — it is a template and would overwrite both live secrets with
placeholders.
DELIBERATELY NO FALLBACK: if shorashim-migrate-secrets is absent the
pod stops at CreateContainerConfigError. Loud is correct here. A
fallback to the application role would reintroduce 42501 at a moment
when everything else looked healthy.
NOT DONE BY THE BUILDER: the secret itself. It carries a live DDL
credential and it is cluster state; both are the kohen's hands, not
mine. The command is in deploy/k8s/secrets.yaml.
VERIFIED:
- kubectl apply --dry-run=client on deployment.yaml, configmap.yaml
and secrets.yaml — all four objects valid (client-side only; no
cluster state touched)
- env/envFrom wiring re-read from the parsed manifest: migrate has
envFrom [shorashim-config, shorashim-secrets] + env DATABASE_URL
from shorashim-migrate-secrets; vav-pipeline has envFrom
[shorashim-config, shorashim-secrets] and no env override
- Doorpost.verify_identity() :ok — no @spec_files entry touched, so
no @expected_hash change
- l1/integrity.sha256 unchanged: 99bbd07a…4ea377
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
91 lines
3.7 KiB
YAML
91 lines
3.7 KiB
YAML
# Shorashim Secrets — template only
|
|
# NEVER commit actual secret values to git.
|
|
# Apply with: kubectl apply -f - after filling in values.
|
|
#
|
|
# The DATABASE_URL connects to Exoscale PostgreSQL (temple-council-db)
|
|
# in ch-gva-2 zone. Retrieve credentials from:
|
|
# exo dbaas show temple-council-db --zone ch-gva-2
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: shorashim-secrets
|
|
namespace: shorashim
|
|
type: Opaque
|
|
stringData:
|
|
# PostgreSQL connection for L3 Oral Tradition — the APPLICATION role.
|
|
# Format: ecto://user:password@host:port/database?sslmode=require
|
|
# This is shorashim_app: INSERT and SELECT on the L3 tables, no DDL,
|
|
# no CREATE on schema public. Append-only is enforced by the grants,
|
|
# not by the application's good behaviour.
|
|
DATABASE_URL: "REPLACE_WITH_ACTUAL_DATABASE_URL"
|
|
|
|
---
|
|
# ============================================================
|
|
# shorashim-migrate-secrets — the DDL role
|
|
# ============================================================
|
|
#
|
|
# KOHEN.md:184 — "the migration runs as avnadmin; the application runs
|
|
# as shorashim_app." TWO ROLES, TWO SECRETS. Only the `migrate` init
|
|
# container reads this one (deployment.yaml overrides DATABASE_URL for
|
|
# that container alone); every app container keeps shorashim-secrets
|
|
# above.
|
|
#
|
|
# WHY IT IS SEPARATE AND NOT JUST A GRANT: Ecto.Migrator.run/3 calls
|
|
# ensure_schema_migrations_table! BEFORE checking what is pending, so
|
|
# it issues CREATE TABLE IF NOT EXISTS schema_migrations on EVERY pod
|
|
# start, even with migrations current. As shorashim_app that is
|
|
# ERROR 42501 permission denied for schema public, and the pod never
|
|
# reaches Running. The alternative — granting shorashim_app CREATE on
|
|
# schema public — collapses the app/migration boundary the runbook
|
|
# draws deliberately. Do not take it.
|
|
#
|
|
# THIS FILE IS A TEMPLATE. Never commit a real value. Create the secret
|
|
# from the live database credential instead:
|
|
#
|
|
# export KUBECONFIG=~/.kube/exoscale-temple-council.yaml
|
|
# umask 077
|
|
# TMP=$(mktemp)
|
|
#
|
|
# exo dbaas user reveal-secrets temple-council-db avnadmin \
|
|
# --zone ch-gva-2 -O json \
|
|
# | python3 -c '
|
|
# import base64, json, subprocess, sys, urllib.parse
|
|
# avn = json.load(sys.stdin)["password"]
|
|
# app = subprocess.run(
|
|
# ["kubectl", "get", "secret", "shorashim-secrets", "-n", "shorashim",
|
|
# "-o", "jsonpath={.data.DATABASE_URL}"],
|
|
# capture_output=True, text=True, check=True).stdout
|
|
# u = urllib.parse.urlsplit(base64.b64decode(app).decode())
|
|
# pw = urllib.parse.quote(avn, safe="")
|
|
# netloc = "avnadmin:%s@%s:%d" % (pw, u.hostname, u.port)
|
|
# sys.stdout.write(
|
|
# urllib.parse.urlunsplit((u.scheme, netloc, u.path, u.query, u.fragment)))
|
|
# ' > "$TMP"
|
|
#
|
|
# kubectl create secret generic shorashim-migrate-secrets \
|
|
# --namespace shorashim --from-file=DATABASE_URL="$TMP"
|
|
#
|
|
# rm -f "$TMP"
|
|
#
|
|
# The rewrite is done in Python, not with `sed s|...|...|`, on purpose.
|
|
# A generated password containing & or | or \ corrupts a sed
|
|
# replacement silently — & re-inserts the matched text and | ends the
|
|
# expression early — and the result is a DATABASE_URL that looks
|
|
# plausible and does not authenticate. quote(safe="") also percent-
|
|
# encodes any character that is not URL-safe in the userinfo field,
|
|
# which sed would have passed through raw. The value goes via a
|
|
# umask-077 temp file rather than --from-literal so the password is
|
|
# never an argv entry visible to `ps`.
|
|
#
|
|
# Absence is meant to be loud: without this secret the pod stops at
|
|
# CreateContainerConfigError. There is no fallback to the app role.
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: shorashim-migrate-secrets
|
|
namespace: shorashim
|
|
type: Opaque
|
|
stringData:
|
|
# ecto://avnadmin:PASSWORD@host:port/database?sslmode=require
|
|
DATABASE_URL: "REPLACE_WITH_AVNADMIN_DATABASE_URL"
|