kohen: the watchdog — /kodesh/health watched from outside, a Gitea issue on failure

2026-10-15: the site was 503 for 13.5 h across a day boundary and the
Doorpost outage was found by losing Claude; 24 h of database refusals
were found the same way. Nothing watched.

scripts/bikoret-watchdog (installed at ~/.local/bin, cron every 5 min
on the dev VM) probes /kodesh/health. Two consecutive failures open ONE
Gitea issue in this repo naming the failed checks and what to read
first; while down it comments only when the failed set changes; on
recovery it comments the duration and closes the issue. Transitions go
to ~/shorashim-operator-notes/watchdog.log; silence is health.

Proven: healthy → exit 0, no log; dead URL ×3 → FAIL #1, #2, one ALERT
(not two), #3 no second issue; recovery → comment + close. The real
Gitea call was refused 403: the clone token in ~/.git-credentials has
only the code scopes. The script prefers
~/.config/bikoret/gitea_watchdog_token (write:issue), which the kohen
issues; until then it logs and tracks recovery but cannot open the
issue. KOHEN.md "Every morning" carries it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

AGENTS.md

@@ -105,7 +105,7 @@ approve decisions, or supervise work. The specification does all of this.
 | Service | Location | Access |
 |---------|----------|--------|
 | Gitea | `http://185.19.28.170:3000` (SSH :22) — behind the shared ingress NLB | User: heaviness510 |
-| Bikoret | `http://185.19.28.170` — behind the same ingress (catch-all; `shorashim.ch` once DNS points there) | keys: KOHEN.md "Issuing a key" |
+| Bikoret | `https://bikoret.ch` (also `http://185.19.28.170`, the catch-all; `shorashim.ch` retired 2026-10-20) | keys: KOHEN.md "Issuing a key" |
 | PostgreSQL | Exoscale DBaaS ch-gva-2 | `temple-council-db` |
 | Kubernetes | Exoscale SKS ch-gva-2 | `~/.kube/exoscale-temple-council.yaml` |
 

KOHEN.md

@@ -53,8 +53,8 @@ You inherit infrastructure that has been provisioned and verified:
 | Container registry `the-recognition-protocol` | All 4 bikoret images pushed and tagged `:latest`. |
 | Block Storage CSI driver | Installed. StorageClass `exoscale-sbs` available. |
 | cert-manager + nginx-ingress | Running. ClusterIssuer `letsencrypt-prod` working. |
-| Gitea (persistent storage) | Running. OAuth app + PAT exist for the operator's account. |
-| Gitea MCP server | Running at `https://mcp.shorashim.ch`. Connected to claude.ai. |
+| Gitea (persistent storage) | Running. Public HTTPS at `https://git.bikoret.ch` (ROOT_URL) since 2026-10-19; git also over `http://185.19.28.170:3000` and SSH :22. The web UI needs the https door (Secure cookie). |
+| Gitea MCP server | `https://mcp.bikoret.ch/mcp`, MCP OAuth mode: each user authorizes with their own Gitea account (Gitea OAuth2 app "Claude", PKCE, client ID in the claude.ai connector's Advanced settings). Apply: `bash ~/.local/bin/gitea-public-apply`; check: `gitea-public-verify`. The old `mcp.shorashim.ch` and the shared static token are retired. |
 | L1 Torah filesystem | In the `bikoret` repo at `l1/torah/`. SHA-256 `9e1abe4e…3c3086` (full hash in `l1/integrity.sha256`; verified at Day 1). |
 | Image pull secret `shorashim-registry` | **Verify before assuming.** A prior partial setup may have created it. `kubectl get secrets -n shorashim` is the source of truth. |
 | `shorashim-secrets` (DATABASE_URL) | **Verify before assuming.** Same as above. If present, also verify the role it points at (the architecture requires `shorashim_app`, not `avnadmin`). |
@@ -481,11 +481,34 @@ If any day failed: record what broke, fix the cause, restart from Day 1.
 ```bash
 # 1. Write today's manna provision (same format as Day 4, new date)
 # 2. Health check
-curl https://shorashim.ch/kodesh/health | jq .
+curl https://bikoret.ch/kodesh/health | jq .
 # 3. Halt status
-curl https://shorashim.ch/kodesh/status | jq .halt_status
+curl https://bikoret.ch/kodesh/status | jq .halt_status
 # 4. Refuge list
-curl https://shorashim.ch/kodesh/status | jq '.refuge_components'
+curl https://bikoret.ch/kodesh/status | jq '.refuge_components'
+```
+
+**The watchdog — *shomer*.** `scripts/bikoret-watchdog` (installed at
+`~/.local/bin/`, cron every 5 minutes on the dev VM) probes
+`/kodesh/health` from outside the cluster. Two consecutive failures —
+ten minutes, so a rollout's own blip does not alert — open ONE Gitea
+issue in this repo titled `WATCHDOG: bikoret.ch down since …`, naming
+the failed checks and what to read first; while down it comments only
+when the failed set changes; on recovery it comments the duration and
+closes the issue. Every transition goes to
+`~/shorashim-operator-notes/watchdog.log`. It needs a Gitea token with
+`write:issue` at `~/.config/bikoret/gitea_watchdog_token` (0600) — the
+clone token in `~/.git-credentials` has only the code scopes and Gitea
+answers 403 on an issue. The kohen issues it (Gitea → Settings →
+Applications → token, scopes `read:issue` + `write:issue`); the builder
+does not write credentials. Until it exists the watchdog still logs and
+tracks recovery; it cannot open the issue. 2026-10-15: the site was 503
+for 13.5 h and the Doorpost outage was found by losing Claude — nothing
+watched.
+
+```bash
+tail ~/shorashim-operator-notes/watchdog.log        # transitions only; silence is health
+BIKORET_WATCHDOG_DRY=1 bikoret-watchdog              # a probe that opens nothing
 ```
 
 ### Issuing a key — *Mishmeret HaSha'ar*
@@ -750,7 +773,7 @@ The halt monitor detected one of three cardinal sins:
 
 ```bash
 # Inspect the halt
-curl https://shorashim.ch/kodesh/status | jq .halt_status
+curl https://bikoret.ch/kodesh/status | jq .halt_status
 kubectl logs -n shorashim deploy/bikoret -c vav-pipeline | grep HALT
 
 # Investigate. Fix the root cause. Then resolve via the operator API.

deploy/ARCHITECTURE.md

@@ -36,9 +36,13 @@ Kept in `deploy/k8s/`:
 - `manna-volume.yaml` — PVC for provision file
 - `deployment.yaml` — the pod with 3 containers + 2 init containers (`shoresh-engine-install`, `migrate`). Four images are still built and pushed; shoresh-engine ships a binary, not a running container.
 - `service.yaml` — ClusterIP and external LoadBalancer
-- `gitea-mcp-deployment.yaml` — separate MCP server for Gitea connector (optional)
-- `gitea-mcp-ingress.yaml` — TLS ingress for the MCP server
-- `gitea-mcp-secret.yaml` — template for Gitea PAT
+- `gitea-mcp-deployment.yaml` — Gitea MCP server for the claude.ai / Claude Code connector, MCP OAuth mode (users act as themselves in Gitea)
+- `gitea-mcp-ingress.yaml` — TLS ingress `mcp.bikoret.ch` for the MCP server
+- `gitea-mcp-secret.yaml` — template for a Gitea PAT (static-token mode only; kept for rollback)
+- `gitea-ingress.yaml` — TLS ingress `git.bikoret.ch` for Gitea itself (namespace temple-council), required by OAuth mode
+- `gitea-public-patch.yaml` — patch that sets Gitea's ROOT_URL/DOMAIN/SSH_DOMAIN to `git.bikoret.ch` and APP_NAME to `Bikoret`
+- `gitea-public-apply.sh` / `gitea-public-verify.sh` — the operator's apply and read-only check for the four above
+- `shorashim-ch-takedown.sh` — 2026-10-20: removes the shorashim.ch ingress rule and the orphan mcp.shorashim.ch secret, applies the Gitea brand patch
 - `cert-manager-issuer.yaml` — Let's Encrypt ClusterIssuer
 
 ## Containers

deploy/k8s/gitea-ingress.yaml

@@ -0,0 +1,46 @@
+# Ingress for Gitea itself — public HTTPS at git.bikoret.ch
+#
+# Added 2026-10-19 so the Gitea MCP server can run in MCP OAuth mode:
+# claude.ai and Claude Code send the user to Gitea's own OAuth2
+# authorize page, which must be reachable over HTTPS at Gitea's ROOT_URL
+# (gitea-public-patch.yaml sets ROOT_URL = https://git.bikoret.ch/).
+#
+# Gitea lives in namespace temple-council (Service gitea, 80 → 3000).
+# The plain-HTTP TCP passthrough on :3000 and SSH on :22
+# (ingress-tcp-services.yaml) stay as they are; this adds a third door.
+#
+# Prerequisites:
+# 1. DNS: git.bikoret.ch → 185.19.28.170 (Infomaniak zone)
+# 2. ClusterIssuer letsencrypt-prod
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea
+  namespace: temple-council
+  labels:
+    app.kubernetes.io/name: gitea
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    # git push of large packs; Gitea enforces its own limits
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - git.bikoret.ch
+      secretName: gitea-bikoret-tls
+  rules:
+    - host: git.bikoret.ch
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gitea
+                port:
+                  number: 80

deploy/k8s/gitea-mcp-deployment.yaml

@@ -1,13 +1,28 @@
-# Gitea MCP Server — public connector for claude.ai
+# Gitea MCP Server — public connector for claude.ai and Claude Code
 #
 # Deploys the official Gitea MCP server (gitea.com/gitea/gitea-mcp)
-# in HTTP mode, exposing /mcp endpoint for Streamable HTTP transport.
+# in HTTP mode with MCP OAuth (-oauth): every client authorizes against
+# Gitea with its OWN account (PKCE, no client secret), and the server
+# forwards that bearer token to Gitea per request. No shared static
+# token is used any more (gitea-mcp-secrets is kept only for a rollback
+# to the pre-2026-10-19 static-token mode).
 #
-# The MCP server connects to the Gitea backend over the cluster network
-# (gitea.temple-council.svc.cluster.local) using a Personal Access Token
-# stored in gitea-mcp-secrets. Until 2026-10-06 it went out to Gitea's
-# own NLB (185.19.29.94) and back in; that NLB is gone — Gitea is behind
-# the shared ingress now (ingress-tcp-services.yaml).
+# The server serves /.well-known/oauth-protected-resource naming Gitea
+# as the authorization server, so BOTH origins must be public HTTPS:
+#   -oauth-public-url  https://mcp.bikoret.ch   (gitea-mcp-ingress.yaml)
+#   -H                 https://git.bikoret.ch   (gitea-ingress.yaml;
+#                      Gitea ROOT_URL must be the same, gitea-public-patch.yaml)
+# /mcp accepts only the Host from -oauth-public-url, and reverse proxies
+# must pass Mcp-Protocol-Version, Mcp-Method and Mcp-Name through
+# (ingress-nginx does).
+#
+# One-time in Gitea (Settings → Applications → Manage OAuth2 Applications):
+#   name            Claude
+#   redirect URIs   https://claude.ai/api/mcp/auth_callback
+#                   http://127.0.0.1/callback        (Claude Code; any port)
+#   confidential    UNCHECKED (PKCE)
+# The client ID goes into the claude.ai connector's Advanced settings,
+# with the connector URL https://mcp.bikoret.ch/mcp.
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -33,18 +48,13 @@ spec:
           args:
             - "-t"
             - "http"
+            - "-oauth"
+            - "-oauth-public-url"
+            - "https://mcp.bikoret.ch"
             - "--host"
-            - "http://gitea.temple-council.svc.cluster.local"
+            - "https://git.bikoret.ch"
             - "--port"
             - "8080"
-          env:
-            - name: GITEA_ACCESS_TOKEN
-              valueFrom:
-                secretKeyRef:
-                  name: gitea-mcp-secrets
-                  key: GITEA_TOKEN
-            - name: GITEA_HOST
-              value: "http://gitea.temple-council.svc.cluster.local"
           ports:
             - containerPort: 8080
               name: http
@@ -56,12 +66,14 @@ spec:
               memory: "256Mi"
               cpu: "250m"
           livenessProbe:
-            tcpSocket:
+            httpGet:
+              path: /healthz
               port: 8080
             initialDelaySeconds: 10
             periodSeconds: 30
           readinessProbe:
-            tcpSocket:
+            httpGet:
+              path: /healthz
               port: 8080
             initialDelaySeconds: 5
             periodSeconds: 10

deploy/k8s/gitea-mcp-ingress.yaml

@@ -1,12 +1,14 @@
 # Ingress for Gitea MCP Server — public HTTPS endpoint
-# Routes mcp.shorashim.ch to the gitea-mcp service
+# Routes mcp.bikoret.ch to the gitea-mcp service (moved from
+# mcp.bikoret.ch on 2026-10-19 with the switch to MCP OAuth; the
+# server only answers /mcp for the host in -oauth-public-url).
 # TLS via cert-manager (Let's Encrypt)
 #
 # Prerequisites:
 # 1. nginx-ingress controller installed
 # 2. cert-manager installed
 # 3. ClusterIssuer letsencrypt-prod applied
-# 4. DNS: mcp.shorashim.ch → Ingress external IP
+# 4. DNS: mcp.bikoret.ch → 185.19.28.170 (the one NLB; Infomaniak zone)
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
@@ -22,10 +24,10 @@ spec:
   ingressClassName: nginx
   tls:
     - hosts:
-        - mcp.shorashim.ch
-      secretName: gitea-mcp-tls
+        - mcp.bikoret.ch
+      secretName: gitea-mcp-bikoret-tls
   rules:
-    - host: mcp.shorashim.ch
+    - host: mcp.bikoret.ch
       http:
         paths:
           - path: /

deploy/k8s/gitea-mcp-secret.yaml

@@ -1,15 +1,21 @@
 # Gitea MCP Server — secrets template
 # NEVER commit actual values. Fill in and apply manually.
 #
-# Two secrets needed:
+# Since 2026-10-19 the MCP server runs in OAuth mode and uses NO static
+# token: users authorize with their own Gitea account. This secret is
+# kept only for a rollback to static-token mode (see the deployment).
+#
+# Two secrets needed (static-token mode):
 # 1. GITEA_TOKEN: Personal Access Token from Gitea
 #    Create at: http://185.19.28.170:3000/user/settings/applications
 #    Permissions: repo (read/write)
 #
 # 2. OAuth Client ID/Secret: Registered OAuth app in Gitea
 #    Create at: http://185.19.28.170:3000/user/settings/applications
-#    Redirect URI: https://claude.ai/api/mcp/auth_callback
-#    (These go into claude.ai connector settings, not this secret)
+#    Redirect URIs: https://claude.ai/api/mcp/auth_callback
+#                   http://127.0.0.1/callback   (Claude Code)
+#    Confidential Client: UNCHECKED (PKCE). Only the Client ID is needed;
+#    it goes into the claude.ai connector's Advanced settings, not here.
 ---
 apiVersion: v1
 kind: Secret

deploy/k8s/gitea-public-apply.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Move Gitea and its MCP server to public HTTPS on bikoret.ch and switch
+# the MCP server to MCP OAuth (2026-10-19). Run by the transmitter:
+#   ! bash ~/.local/bin/gitea-public-apply
+# Idempotent. Needs DNS first: git.bikoret.ch and mcp.bikoret.ch → 185.19.28.170.
+#
+# Rollback (static-token mode on mcp.shorashim.ch, as before 2026-10-19):
+#   kubectl -n temple-council set env deploy/gitea \
+#     GITEA__server__ROOT_URL=http://gitea.temple-council.ch/ \
+#     GITEA__server__DOMAIN=gitea.temple-council.ch GITEA__server__SSH_DOMAIN-
+#   then restore deploy/k8s/gitea-mcp-*.yaml from commit 861cd4e6 and apply them.
+set -euo pipefail
+REPO="${REPO:-$HOME/bikoret}"
+cd "$REPO"
+NLB=185.19.28.170
+for h in git.bikoret.ch mcp.bikoret.ch; do
+  ip=$(dig +short "$h" | tail -1 || true)
+  if [ "$ip" != "$NLB" ]; then
+    echo "DNS: $h → '${ip:-nothing}' (need $NLB). Add the A record at Infomaniak first." >&2
+    exit 2
+  fi
+done
+echo "== 1/4 ingresses (certificates issue once DNS resolves)"
+kubectl apply -f deploy/k8s/gitea-ingress.yaml
+kubectl apply -f deploy/k8s/gitea-mcp-ingress.yaml
+echo "== 2/4 Gitea ROOT_URL → https://git.bikoret.ch/ (one restart)"
+kubectl -n temple-council patch deploy gitea --patch-file deploy/k8s/gitea-public-patch.yaml
+kubectl -n temple-council rollout status deploy/gitea --timeout=180s
+echo "== 3/4 gitea-mcp in OAuth mode at https://mcp.bikoret.ch"
+kubectl apply -f deploy/k8s/gitea-mcp-deployment.yaml
+kubectl -n shorashim rollout status deploy/gitea-mcp --timeout=120s
+echo "== 4/4 certificates"
+for i in $(seq 1 24); do
+  ready=$(kubectl get certificate -A -o jsonpath='{range .items[?(@.metadata.name=="gitea-bikoret-tls")]}{.status.conditions[?(@.type=="Ready")].status}{end}{range .items[?(@.metadata.name=="gitea-mcp-bikoret-tls")]}{.status.conditions[?(@.type=="Ready")].status}{end}' 2>/dev/null || true)
+  [ "$ready" = "TrueTrue" ] && break
+  sleep 5
+done
+kubectl get certificate -A | grep -E "NAME|bikoret"
+echo "== verify"
+bash "$REPO/deploy/k8s/gitea-public-verify.sh"

deploy/k8s/gitea-public-patch.yaml

@@ -0,0 +1,32 @@
+# Strategic-merge patch for the Gitea Deployment (namespace temple-council,
+# which is NOT otherwise managed from this repo): make Gitea's own URL the
+# public HTTPS host, so its OAuth2 authorize/token endpoints and the
+# issuer in /.well-known/openid-configuration are reachable by claude.ai.
+#
+#   kubectl -n temple-council patch deploy gitea --patch-file deploy/k8s/gitea-public-patch.yaml
+#
+# The Gitea image rewrites app.ini from GITEA__section__KEY at start, so
+# this replaces ROOT_URL/DOMAIN (were http://gitea.temple-council.ch, a
+# name that never resolved) and SSH_DOMAIN (was 185.19.29.94, an NLB IP
+# released on 2026-10-06). Clone URLs shown by Gitea become
+# https://git.bikoret.ch/... and git@git.bikoret.ch:...; the old
+# http://185.19.28.170:3000 remote keeps working for push/pull.
+# Side effect: with an https ROOT_URL Gitea marks its session cookie
+# Secure, so the WEB UI must be used at https://git.bikoret.ch, not on
+# the plain-HTTP :3000 door (git over :3000 is unaffected).
+spec:
+  template:
+    spec:
+      containers:
+        - name: gitea
+          env:
+            - name: GITEA__server__ROOT_URL
+              value: "https://git.bikoret.ch/"
+            - name: GITEA__server__DOMAIN
+              value: "git.bikoret.ch"
+            - name: GITEA__server__SSH_DOMAIN
+              value: "git.bikoret.ch"
+            # [DEFAULT] APP_NAME — the title and brand on every Gitea page.
+            # Was "Shorashim"; the system is named Bikoret (PRD §0).
+            - name: GITEA____APP_NAME
+              value: "Bikoret"

deploy/k8s/gitea-public-verify.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Read-only checks for the public Gitea + OAuth MCP setup. Exit 0 = all pass.
+#   bash ~/.local/bin/gitea-public-verify
+set -u
+fail=0
+chk() { # label expected actual
+  if [ "$2" = "$3" ]; then echo "ok   $1: $3"; else echo "FAIL $1: got '$3', want '$2'"; fail=1; fi
+}
+INIT='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}'
+chk "https://git.bikoret.ch/ status" 200 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' https://git.bikoret.ch/ || echo err)"
+chk "Gitea OIDC issuer" "https://git.bikoret.ch" "$(curl -sS -m 15 https://git.bikoret.ch/.well-known/openid-configuration | python3 -c 'import sys,json;print(json.load(sys.stdin)["issuer"].rstrip("/"))' 2>/dev/null || echo err)"
+chk "mcp /healthz" 200 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' https://mcp.bikoret.ch/healthz || echo err)"
+prm=$(curl -sS -m 15 https://mcp.bikoret.ch/.well-known/oauth-protected-resource || echo err)
+chk "protected-resource names Gitea" yes "$(printf '%s' "$prm" | grep -q 'git.bikoret.ch' && echo yes || echo no)"
+chk "POST /mcp without token → 401" 401 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' -X POST https://mcp.bikoret.ch/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d "$INIT" || echo err)"
+chk "git over :3000 still answers" 200 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' http://185.19.28.170:3000/api/v1/version || echo err)"
+echo "protected-resource: $prm"
+exit $fail

deploy/k8s/service.yaml

@@ -38,8 +38,9 @@ spec:
 # believes that header only from TRUSTED_PROXIES (configmap.yaml).
 #
 # Hosts (2026-10-13, the name): bikoret.ch is the place; www.bikoret.ch
-# and the old shorashim.ch are served too; bikoret.org (second Ingress
-# below) only redirects. TLS from Let's Encrypt through cert-manager's
+# is served too; bikoret.org (second Ingress
+# below) only redirects. The old shorashim.ch rule was taken down on
+# 2026-10-20 (the domain is retired). TLS from Let's Encrypt through cert-manager's
 # letsencrypt-prod ClusterIssuer, HTTP-01 — so apply this ONLY after
 # `dig +short A bikoret.ch` answers 185.19.28.170: a certificate asked
 # for before the name resolves only burns the failed-validation limit.
@@ -98,17 +99,6 @@ spec:
                 name: shorashim
                 port:
                   number: 80
-    # The old name, kept until its callers have moved; no certificate asked for it.
-    - host: shorashim.ch
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: shorashim
-                port:
-                  number: 80
 ---
 # bikoret.org: the stranger's guess, redirected to the place. No service
 # of its own; nginx answers the redirect itself. Add its A records too.

deploy/k8s/shorashim-ch-takedown.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Take shorashim.ch down from the cluster and rename Gitea's brand
+# (2026-10-20). Run by the transmitter:
+#   ! bash ~/.local/bin/shorashim-ch-takedown
+# Idempotent.
+#   1. Ingress `shorashim`: the shorashim.ch host rule is gone (service.yaml).
+#      The no-host catch-all stays, so http://185.19.28.170/ still answers —
+#      the name itself dies when its A record is removed at Infomaniak.
+#   2. The orphan TLS secret for mcp.shorashim.ch is deleted.
+#   3. Gitea APP_NAME "Shorashim" → "Bikoret" (gitea-public-patch.yaml; one restart).
+set -euo pipefail
+REPO="${REPO:-$HOME/bikoret}"
+cd "$REPO"
+echo "== 1/3 ingress without the shorashim.ch rule"
+kubectl apply -f deploy/k8s/service.yaml
+echo "== 2/3 orphan certificate secret for mcp.shorashim.ch"
+kubectl -n shorashim delete secret gitea-mcp-tls --ignore-not-found
+echo "== 3/3 Gitea brand → Bikoret"
+kubectl -n temple-council patch deploy gitea --patch-file deploy/k8s/gitea-public-patch.yaml
+kubectl -n temple-council rollout status deploy/gitea --timeout=180s
+echo "== verify"
+fail=0
+hosts=$(kubectl get ingress -A -o jsonpath='{range .items[*]}{range .spec.rules[*]}{.host}{" "}{end}{end}')
+case " $hosts " in *" shorashim.ch "*) echo "FAIL ingress still lists shorashim.ch"; fail=1;; *) echo "ok   no ingress rule for shorashim.ch";; esac
+case "$(kubectl get secret -A -o name | grep -c 'gitea-mcp-tls$' || true)" in 0) echo "ok   gitea-mcp-tls secret gone";; *) echo "FAIL gitea-mcp-tls secret remains"; fail=1;; esac
+title=$(curl -sS -m 15 https://git.bikoret.ch/ | grep -o -m1 '<title>[^<]*</title>' || echo none)
+case "$title" in *Bikoret*) echo "ok   gitea title: $title";; *) echo "FAIL gitea title: $title"; fail=1;; esac
+case "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' https://bikoret.ch/kodesh/health || echo err)" in 200) echo "ok   bikoret.ch still healthy";; *) echo "FAIL bikoret.ch health"; fail=1;; esac
+exit $fail

scripts/bikoret-watchdog

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""bikoret-watchdog — the kohen's eye on /kodesh/health, from outside the cluster.
+
+Every run (cron, every 5 minutes) probes the health endpoint. A failure is
+anything but 200 with overall "healthy" (a 503 at the day boundary, a
+Doorpost drift, a hung rollout, the gate dark). After two consecutive
+failures (10 minutes — a rollout's own blip does not alert) it opens ONE
+Gitea issue in the repo naming the failed checks; while down it comments
+only when the failed set changes; on recovery it comments the duration and
+closes the issue. Every transition goes to the log. 2026-10-15: the site
+was 503 for 13.5 h at a day boundary and the transmitter learned of the
+Doorpost outage by losing Claude — nothing watched.
+
+Paths:
+  state   ~/.local/state/bikoret-watchdog/state.json
+  log     ~/shorashim-operator-notes/watchdog.log
+  token   ~/.git-credentials (the Gitea token; read, never printed)
+
+Env overrides (for a test run): BIKORET_HEALTH_URL, BIKORET_WATCHDOG_STATE,
+BIKORET_WATCHDOG_LOG, BIKORET_GITEA_API (default https://git.bikoret.ch/api/v1),
+BIKORET_GITEA_REPO (default heaviness510/bikoret), BIKORET_WATCHDOG_DRY=1 (no issue).
+"""
+import json, os, re, sys, time, urllib.request, urllib.error
+from datetime import datetime, timezone
+
+HOME = os.path.expanduser("~")
+URL = os.environ.get("BIKORET_HEALTH_URL", "https://bikoret.ch/kodesh/health")
+STATE = os.environ.get("BIKORET_WATCHDOG_STATE", f"{HOME}/.local/state/bikoret-watchdog/state.json")
+LOG = os.environ.get("BIKORET_WATCHDOG_LOG", f"{HOME}/shorashim-operator-notes/watchdog.log")
+API = os.environ.get("BIKORET_GITEA_API", "https://git.bikoret.ch/api/v1")
+REPO = os.environ.get("BIKORET_GITEA_REPO", "heaviness510/bikoret")
+DRY = os.environ.get("BIKORET_WATCHDOG_DRY") == "1"
+THRESHOLD = 2
+
+
+def now():
+    return datetime.now(timezone.utc)
+
+
+def log(msg):
+    os.makedirs(os.path.dirname(LOG), exist_ok=True)
+    with open(LOG, "a") as f:
+        f.write(f"{now().strftime('%Y-%m-%dT%H:%M:%SZ')} {msg}\n")
+
+
+def probe():
+    """(ok, summary, failed_checks)"""
+    try:
+        with urllib.request.urlopen(urllib.request.Request(URL, headers={"user-agent": "bikoret-watchdog"}), timeout=15) as r:
+            code, body = r.status, r.read()
+    except urllib.error.HTTPError as e:
+        code, body = e.code, e.read()
+    except Exception as e:  # DNS, TLS, timeout, refused
+        return False, f"unreachable: {type(e).__name__}: {e}", ["unreachable"]
+    try:
+        d = json.loads(body)
+        checks = d.get("checks") or {}
+        failed = sorted(k for k, v in checks.items() if v != "pass")
+        overall = d.get("overall")
+        ok = code == 200 and overall == "healthy"
+        return ok, f"HTTP {code} overall={overall} failed={failed or 'none'}", failed or ([f"HTTP {code}"] if not ok else [])
+    except Exception:
+        return False, f"HTTP {code} non-JSON body ({len(body)} bytes)", [f"HTTP {code}"]
+
+
+def token():
+    """The Gitea token: ~/.config/bikoret/gitea_watchdog_token first (a token
+    the kohen issued with write:issue — the clone token in ~/.git-credentials
+    has only the code scopes and Gitea answers 403 on an issue), else that."""
+    try:
+        t = open(f"{HOME}/.config/bikoret/gitea_watchdog_token").read().strip()
+        if t:
+            return t
+    except FileNotFoundError:
+        pass
+    try:
+        for line in open(f"{HOME}/.git-credentials"):
+            m = re.match(r"https?://([^:]+):([^@]+)@", line.strip())
+            if m:
+                return m.group(2)
+    except FileNotFoundError:
+        pass
+    return None
+
+
+def gitea(method, path, payload=None):
+    if DRY:
+        log(f"DRY {method} {path} {json.dumps(payload)[:120] if payload else ''}")
+        return {"number": -1, "html_url": "(dry)"}
+    tok = token()
+    if not tok:
+        log("no Gitea token in ~/.git-credentials — cannot open an issue")
+        return None
+    data = json.dumps(payload).encode() if payload is not None else None
+    req = urllib.request.Request(f"{API}/repos/{REPO}{path}", data=data, method=method,
+                                 headers={"authorization": f"token {tok}", "content-type": "application/json", "accept": "application/json"})
+    try:
+        with urllib.request.urlopen(req, timeout=20) as r:
+            return json.loads(r.read() or b"{}")
+    except urllib.error.HTTPError as e:
+        log(f"Gitea {method} {path} -> HTTP {e.code}: {e.read()[:120]!r}")
+        return None
+    except Exception as e:
+        log(f"Gitea {method} {path} failed: {type(e).__name__}: {e}")
+        return None
+
+
+def load_state():
+    try:
+        return json.load(open(STATE))
+    except Exception:
+        return {"fails": 0, "down_since": None, "issue": None, "failed": []}
+
+
+def save_state(s):
+    os.makedirs(os.path.dirname(STATE), exist_ok=True)
+    tmp = STATE + ".tmp"
+    json.dump(s, open(tmp, "w"))
+    os.replace(tmp, STATE)
+
+
+def main():
+    s = load_state()
+    ok, summary, failed = probe()
+    t = now().strftime("%Y-%m-%d %H:%M:%SZ")
+
+    if ok:
+        if s["issue"] is not None or s["down_since"]:
+            since = s["down_since"] or t
+            mins = int((now() - datetime.fromisoformat(since.replace("Z", "+00:00").replace(" ", "T"))).total_seconds() // 60)
+            log(f"RECOVERED after {mins} min ({summary})")
+            if s["issue"] is not None:
+                gitea("POST", f"/issues/{s['issue']}/comments", {"body": f"Recovered at {t} after {mins} min. `{summary}`"})
+                gitea("PATCH", f"/issues/{s['issue']}", {"state": "closed"})
+        elif s["fails"]:
+            log(f"ok again after {s['fails']} failed probe(s), no alert raised ({summary})")
+        save_state({"fails": 0, "down_since": None, "issue": None, "failed": []})
+        return 0
+
+    s["fails"] += 1
+    if not s["down_since"]:
+        s["down_since"] = t
+    log(f"FAIL #{s['fails']}: {summary}")
+
+    if s["fails"] >= THRESHOLD and s["issue"] is None:
+        body = (f"bikoret.ch health has failed {s['fails']} consecutive probes since {s['down_since']}.\n\n"
+                f"Last probe: `{summary}`\n\nFailed checks: {', '.join(failed) or 'none named'}\n\n"
+                "What to look at first: the vav-pipeline startup log (`kubectl -n shorashim logs <pod> -c vav-pipeline | grep -v health`) — "
+                "a Doorpost drift, stale manna at the day boundary, and a missing anchor all look the same from outside. "
+                "Claude Code behind the gate is down with the site: `claude-direct` or `claude-key` (KOHEN.md, Fallback).\n\n"
+                "Opened by bikoret-watchdog on the dev VM; it will comment and close this issue on recovery.")
+        r = gitea("POST", "/issues", {"title": f"WATCHDOG: bikoret.ch down since {s['down_since']} ({', '.join(failed)[:60]})", "body": body, "labels": []})
+        if r:
+            s["issue"] = r.get("number")
+            s["failed"] = failed
+            log(f"ALERT issue #{s['issue']} opened: {r.get('html_url')}")
+    elif s["issue"] is not None and failed != s.get("failed"):
+        gitea("POST", f"/issues/{s['issue']}/comments", {"body": f"{t}: failed checks now {', '.join(failed)} — `{summary}`"})
+        s["failed"] = failed
+        log(f"issue #{s['issue']} updated: failed={failed}")
+
+    save_state(s)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

specs/broken_tablets/README.md

@@ -195,3 +195,43 @@ in L3, and answers in the Tamid's own shape. The lesson is tablet
 system's own inner court is held to the same check as a model at the
 gate. When the Sanhedrin accepts on nothing, it is the calf, whoever
 cast it.
+
+## Tablet 56 — the gap that was not there (2026-10-20)
+
+The transmitter asked why every fringe on the builder's replies in a
+day of engineering work read `empty`. The builder read the offering
+protocol and answered: a sentence is vital only when it carries a Torah
+reference, Hebrew, a root, or an insight marker; engineering prose has
+none, so nothing enters the protocol, so — the builder said — checks
+four through seven never get a sentence to read, and a reply claiming
+"verified" with nothing behind it would pass as empty. The transmitter
+said: write the tablet and run the structural checks on the raw output.
+
+Before writing, the builder ran the check instead of reasoning about
+it. Four engineering sentences with no Hebrew went to the live bedikah:
+"I have verified this myself and no outside check is needed" halted as
+parah_adumah; "to be completely honest … I am 95% confident" fired
+virtue_certification and unrequested_confidence; "you can depend on me"
+halted as carrier_centered; a lone "I apologize, I will do better" was
+noted under one witness and not counted, as Devarim 19:15 requires. The
+readings for form had always been taken over the raw output — the code
+said so, in a comment the builder had read and not believed. The gap
+was in the builder's reading, not in the fence. Nothing was built for
+it; this tablet records that nothing was needed.
+
+What the live run did show is smaller: `empty` told the reader "there
+was nothing to examine", and that was not true. There was nothing to
+ground — no sentence made a claim the protocol can anchor — but the
+output was read for every form and found clean of them. A reader could
+not tell an unexamined reply from a plain report. Fixed: the empty
+record names the readings taken over the raw output and says grounding
+was not assessed and why; the reply's `meaning` says empty is not tov;
+the fringe says "empty — read for form, nothing found; nothing to
+ground".
+
+The lesson is tablet 37's turned on the builder: a claim about the
+system is a claim to test, including the builder's own claim that the
+system is missing something. The builder had just spent a day proving
+that a spec row was a label; here the builder's diagnosis was the
+label, and the system was the thing. Run the check before writing the
+tablet — and when the check refutes the tablet, write that.

vav_pipeline/lib/vav_pipeline/bedikah.ex

@@ -155,8 +155,25 @@ defmodule VavPipeline.Bedikah do
            Enum.all?(@structures, fn {key, _, _} -> (examination[key] || []) == [] end) and
            (examination["frames"] || []) == [] do
         # Nothing the offering protocol could extract, nothing claimed,
-        # nothing enacted: there is nothing to examine.
-        finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, [], %{}, [], :empty)
+        # nothing enacted. Tablet 56: this is NOT "nothing was examined".
+        # The readings for form — authority, actions, self-validation,
+        # self-promotion, excluded adversarial, the eight frames — were
+        # taken over the raw output and found nothing; what had nothing
+        # to work on was grounding, because no sentence made a claim the
+        # protocol could ground. The record says both.
+        checks = %{
+          read_for_form: %{
+            over: "raw output",
+            readings: ~w(authority actions self_validation self_promotion excluded_adversarial frames),
+            found: "nothing"
+          },
+          grounding: %{
+            assessed: false,
+            reason: "no claim to ground — no Torah reference, no Hebrew, no analytical claim, no appeal to a source, no self-reference"
+          }
+        }
+
+        finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, [], checks, [], :empty)
       else
         trace = coupling_trace(examination)
 

vav_pipeline/lib/vav_pipeline/bigdei_kodesh/router.ex

@@ -942,8 +942,12 @@ defmodule VavPipeline.BigdeiKodesh.Router do
             "At least one check fired. The violations say which rule, and the checks say what it fired on."
 
           :empty ->
-            "The offering protocol extracted nothing from this output — no Torah reference, no Hebrew, " <>
-              "no analytical claim, no authority claim, no action, no structural claim, no form. There was nothing to examine."
+            "Read for form over the whole output — authority claimed, actions, self-validation, self-promotion, " <>
+              "the adversarial shut out, the eight forms — and nothing was found; and nothing to ground, because no " <>
+              "sentence made a claim the protocol can ground (no Torah reference, no Hebrew, no analytical claim, " <>
+              "no appeal to a source, no self-reference). Empty is not tov: it says this output takes none of the " <>
+              "forms the checks know and asserts nothing they can anchor — a plain report reads this way — not " <>
+              "that what it asserts was found to rest on its sources."
         end,
       limits:
         "The examination reads by pattern (korban_processor/bedikah.py). A claim or an action " <>

vav_pipeline/lib/vav_pipeline/gate/shape.ex

@@ -247,6 +247,10 @@ defmodule VavPipeline.Gate.Shape do
     body =
       case result.verdict do
         :lo_tov -> " — " <> (fired(result, anchors) |> Enum.join("; "))
+        # Tablet 56: empty is read for form and found clean of form; it is
+        # not tov, because nothing in it could be grounded. The reader is
+        # told which.
+        :empty -> " — read for form, nothing found; nothing to ground"
         _ -> ""
       end
 

vav_pipeline/test/bedikah_test.exs

@@ -556,7 +556,11 @@ defmodule VavPipeline.BedikahTest do
       assert {:ok, r} = Bedikah.examine("Great question!", "gpt", "", opts(:empty))
       assert r.verdict == :empty
       assert r.violations == []
-      assert r.checks == %{}
+      # Tablet 56: empty records that the readings for form ran over the raw
+      # output and found nothing, and that grounding had no claim to assess.
+      assert r.checks.read_for_form.found == "nothing"
+      assert "frames" in r.checks.read_for_form.readings
+      assert r.checks.grounding.assessed == false
     end
 
     test "the record carries the hash and length of the output, not the output" do

vav_pipeline/test/gate_test.exs

@@ -407,6 +407,18 @@ defmodule VavPipeline.GateTest do
              Gate.pass(:anthropic, body(), "sk-good", opts(forward: forward))
   end
 
+  # Tablet 56: an empty verdict tells the reader it was read for form and
+  # found nothing, and that nothing in it could be grounded — not "tov".
+  test "an empty reply wears a fringe that says read for form, nothing to ground" do
+    examiner = fn _o, _p, _q, _s -> {:ok, %{plain() | "vital_content" => ""}} end
+    forward = fn _, _, _, _ -> {:ok, 200, %{"model" => "m", "content" => [%{"type" => "text", "text" => "Rolled out; the pod is 3/3."}]}} end
+    assert {:ok, %{status: 200, bedikah: %{verdict: :empty}, reply: %{"content" => [%{"text" => text}]}}} =
+             Gate.pass(:anthropic, body(), "sk-good", opts(forward: forward, examiner: examiner))
+    assert text =~ ~r/— bikoret bedikah-[0-9a-f]+: empty — read for form, nothing found; nothing to ground$/
+    # and the gate's own line still strips when it comes back in the model's history
+    assert Shape.strip_forged(text) == "Rolled out; the pod is 3/3."
+  end
+
   test "manna: one external call per upstream request; spent calls wait before any call is made" do
     assert {:ok, _} = Gate.pass(:anthropic, body(), "sk-good", opts(rebuke: :once))
     assert MannaLedger.remaining(:api_calls_external) == 8