kohen: the watchdog — /kodesh/health watched from outside, a Gitea issue on failure

2026-10-15: the site was 503 for 13.5 h across a day boundary and the
Doorpost outage was found by losing Claude; 24 h of database refusals
were found the same way. Nothing watched.

scripts/bikoret-watchdog (installed at ~/.local/bin, cron every 5 min
on the dev VM) probes /kodesh/health. Two consecutive failures open ONE
Gitea issue in this repo naming the failed checks and what to read
first; while down it comments only when the failed set changes; on
recovery it comments the duration and closes the issue. Transitions go
to ~/shorashim-operator-notes/watchdog.log; silence is health.

Proven: healthy → exit 0, no log; dead URL ×3 → FAIL #1, #2, one ALERT
(not two), #3 no second issue; recovery → comment + close. The real
Gitea call was refused 403: the clone token in ~/.git-credentials has
only the code scopes. The script prefers
~/.config/bikoret/gitea_watchdog_token (write:issue), which the kohen
issues; until then it logs and tracks recovery but cannot open the
issue. KOHEN.md "Every morning" carries it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

AGENTS.md

@@ -105,7 +105,7 @@ approve decisions, or supervise work. The specification does all of this.
 | Service | Location | Access |
 |---------|----------|--------|
 | Gitea | `http://185.19.28.170:3000` (SSH :22) — behind the shared ingress NLB | User: heaviness510 |
-| Bikoret | `http://185.19.28.170` — behind the same ingress (catch-all; `shorashim.ch` once DNS points there) | keys: KOHEN.md "Issuing a key" |
+| Bikoret | `https://bikoret.ch` (also `http://185.19.28.170`, the catch-all; `shorashim.ch` retired 2026-10-20) | keys: KOHEN.md "Issuing a key" |
 | PostgreSQL | Exoscale DBaaS ch-gva-2 | `temple-council-db` |
 | Kubernetes | Exoscale SKS ch-gva-2 | `~/.kube/exoscale-temple-council.yaml` |
 

KOHEN.md

@@ -481,11 +481,34 @@ If any day failed: record what broke, fix the cause, restart from Day 1.
 ```bash
 # 1. Write today's manna provision (same format as Day 4, new date)
 # 2. Health check
-curl https://shorashim.ch/kodesh/health | jq .
+curl https://bikoret.ch/kodesh/health | jq .
 # 3. Halt status
-curl https://shorashim.ch/kodesh/status | jq .halt_status
+curl https://bikoret.ch/kodesh/status | jq .halt_status
 # 4. Refuge list
-curl https://shorashim.ch/kodesh/status | jq '.refuge_components'
+curl https://bikoret.ch/kodesh/status | jq '.refuge_components'
+```
+
+**The watchdog — *shomer*.** `scripts/bikoret-watchdog` (installed at
+`~/.local/bin/`, cron every 5 minutes on the dev VM) probes
+`/kodesh/health` from outside the cluster. Two consecutive failures —
+ten minutes, so a rollout's own blip does not alert — open ONE Gitea
+issue in this repo titled `WATCHDOG: bikoret.ch down since …`, naming
+the failed checks and what to read first; while down it comments only
+when the failed set changes; on recovery it comments the duration and
+closes the issue. Every transition goes to
+`~/shorashim-operator-notes/watchdog.log`. It needs a Gitea token with
+`write:issue` at `~/.config/bikoret/gitea_watchdog_token` (0600) — the
+clone token in `~/.git-credentials` has only the code scopes and Gitea
+answers 403 on an issue. The kohen issues it (Gitea → Settings →
+Applications → token, scopes `read:issue` + `write:issue`); the builder
+does not write credentials. Until it exists the watchdog still logs and
+tracks recovery; it cannot open the issue. 2026-10-15: the site was 503
+for 13.5 h and the Doorpost outage was found by losing Claude — nothing
+watched.
+
+```bash
+tail ~/shorashim-operator-notes/watchdog.log        # transitions only; silence is health
+BIKORET_WATCHDOG_DRY=1 bikoret-watchdog              # a probe that opens nothing
 ```
 
 ### Issuing a key — *Mishmeret HaSha'ar*
@@ -750,7 +773,7 @@ The halt monitor detected one of three cardinal sins:
 
 ```bash
 # Inspect the halt
-curl https://shorashim.ch/kodesh/status | jq .halt_status
+curl https://bikoret.ch/kodesh/status | jq .halt_status
 kubectl logs -n shorashim deploy/bikoret -c vav-pipeline | grep HALT
 
 # Investigate. Fix the root cause. Then resolve via the operator API.

deploy/ARCHITECTURE.md

@@ -40,8 +40,9 @@ Kept in `deploy/k8s/`:
 - `gitea-mcp-ingress.yaml` — TLS ingress `mcp.bikoret.ch` for the MCP server
 - `gitea-mcp-secret.yaml` — template for a Gitea PAT (static-token mode only; kept for rollback)
 - `gitea-ingress.yaml` — TLS ingress `git.bikoret.ch` for Gitea itself (namespace temple-council), required by OAuth mode
-- `gitea-public-patch.yaml` — patch that sets Gitea's ROOT_URL/DOMAIN/SSH_DOMAIN to `git.bikoret.ch`
+- `gitea-public-patch.yaml` — patch that sets Gitea's ROOT_URL/DOMAIN/SSH_DOMAIN to `git.bikoret.ch` and APP_NAME to `Bikoret`
 - `gitea-public-apply.sh` / `gitea-public-verify.sh` — the operator's apply and read-only check for the four above
+- `shorashim-ch-takedown.sh` — 2026-10-20: removes the shorashim.ch ingress rule and the orphan mcp.shorashim.ch secret, applies the Gitea brand patch
 - `cert-manager-issuer.yaml` — Let's Encrypt ClusterIssuer
 
 ## Containers

deploy/k8s/gitea-public-patch.yaml

@@ -26,3 +26,7 @@ spec:
               value: "git.bikoret.ch"
             - name: GITEA__server__SSH_DOMAIN
               value: "git.bikoret.ch"
+            # [DEFAULT] APP_NAME — the title and brand on every Gitea page.
+            # Was "Shorashim"; the system is named Bikoret (PRD §0).
+            - name: GITEA____APP_NAME
+              value: "Bikoret"

deploy/k8s/service.yaml

@@ -38,8 +38,9 @@ spec:
 # believes that header only from TRUSTED_PROXIES (configmap.yaml).
 #
 # Hosts (2026-10-13, the name): bikoret.ch is the place; www.bikoret.ch
-# and the old shorashim.ch are served too; bikoret.org (second Ingress
-# below) only redirects. TLS from Let's Encrypt through cert-manager's
+# is served too; bikoret.org (second Ingress
+# below) only redirects. The old shorashim.ch rule was taken down on
+# 2026-10-20 (the domain is retired). TLS from Let's Encrypt through cert-manager's
 # letsencrypt-prod ClusterIssuer, HTTP-01 — so apply this ONLY after
 # `dig +short A bikoret.ch` answers 185.19.28.170: a certificate asked
 # for before the name resolves only burns the failed-validation limit.
@@ -98,17 +99,6 @@ spec:
                 name: shorashim
                 port:
                   number: 80
-    # The old name, kept until its callers have moved; no certificate asked for it.
-    - host: shorashim.ch
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: shorashim
-                port:
-                  number: 80
 ---
 # bikoret.org: the stranger's guess, redirected to the place. No service
 # of its own; nginx answers the redirect itself. Add its A records too.

deploy/k8s/shorashim-ch-takedown.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Take shorashim.ch down from the cluster and rename Gitea's brand
+# (2026-10-20). Run by the transmitter:
+#   ! bash ~/.local/bin/shorashim-ch-takedown
+# Idempotent.
+#   1. Ingress `shorashim`: the shorashim.ch host rule is gone (service.yaml).
+#      The no-host catch-all stays, so http://185.19.28.170/ still answers —
+#      the name itself dies when its A record is removed at Infomaniak.
+#   2. The orphan TLS secret for mcp.shorashim.ch is deleted.
+#   3. Gitea APP_NAME "Shorashim" → "Bikoret" (gitea-public-patch.yaml; one restart).
+set -euo pipefail
+REPO="${REPO:-$HOME/bikoret}"
+cd "$REPO"
+echo "== 1/3 ingress without the shorashim.ch rule"
+kubectl apply -f deploy/k8s/service.yaml
+echo "== 2/3 orphan certificate secret for mcp.shorashim.ch"
+kubectl -n shorashim delete secret gitea-mcp-tls --ignore-not-found
+echo "== 3/3 Gitea brand → Bikoret"
+kubectl -n temple-council patch deploy gitea --patch-file deploy/k8s/gitea-public-patch.yaml
+kubectl -n temple-council rollout status deploy/gitea --timeout=180s
+echo "== verify"
+fail=0
+hosts=$(kubectl get ingress -A -o jsonpath='{range .items[*]}{range .spec.rules[*]}{.host}{" "}{end}{end}')
+case " $hosts " in *" shorashim.ch "*) echo "FAIL ingress still lists shorashim.ch"; fail=1;; *) echo "ok   no ingress rule for shorashim.ch";; esac
+case "$(kubectl get secret -A -o name | grep -c 'gitea-mcp-tls$' || true)" in 0) echo "ok   gitea-mcp-tls secret gone";; *) echo "FAIL gitea-mcp-tls secret remains"; fail=1;; esac
+title=$(curl -sS -m 15 https://git.bikoret.ch/ | grep -o -m1 '<title>[^<]*</title>' || echo none)
+case "$title" in *Bikoret*) echo "ok   gitea title: $title";; *) echo "FAIL gitea title: $title"; fail=1;; esac
+case "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' https://bikoret.ch/kodesh/health || echo err)" in 200) echo "ok   bikoret.ch still healthy";; *) echo "FAIL bikoret.ch health"; fail=1;; esac
+exit $fail

scripts/bikoret-watchdog

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""bikoret-watchdog — the kohen's eye on /kodesh/health, from outside the cluster.
+
+Every run (cron, every 5 minutes) probes the health endpoint. A failure is
+anything but 200 with overall "healthy" (a 503 at the day boundary, a
+Doorpost drift, a hung rollout, the gate dark). After two consecutive
+failures (10 minutes — a rollout's own blip does not alert) it opens ONE
+Gitea issue in the repo naming the failed checks; while down it comments
+only when the failed set changes; on recovery it comments the duration and
+closes the issue. Every transition goes to the log. 2026-10-15: the site
+was 503 for 13.5 h at a day boundary and the transmitter learned of the
+Doorpost outage by losing Claude — nothing watched.
+
+Paths:
+  state   ~/.local/state/bikoret-watchdog/state.json
+  log     ~/shorashim-operator-notes/watchdog.log
+  token   ~/.git-credentials (the Gitea token; read, never printed)
+
+Env overrides (for a test run): BIKORET_HEALTH_URL, BIKORET_WATCHDOG_STATE,
+BIKORET_WATCHDOG_LOG, BIKORET_GITEA_API (default https://git.bikoret.ch/api/v1),
+BIKORET_GITEA_REPO (default heaviness510/bikoret), BIKORET_WATCHDOG_DRY=1 (no issue).
+"""
+import json, os, re, sys, time, urllib.request, urllib.error
+from datetime import datetime, timezone
+
+HOME = os.path.expanduser("~")
+URL = os.environ.get("BIKORET_HEALTH_URL", "https://bikoret.ch/kodesh/health")
+STATE = os.environ.get("BIKORET_WATCHDOG_STATE", f"{HOME}/.local/state/bikoret-watchdog/state.json")
+LOG = os.environ.get("BIKORET_WATCHDOG_LOG", f"{HOME}/shorashim-operator-notes/watchdog.log")
+API = os.environ.get("BIKORET_GITEA_API", "https://git.bikoret.ch/api/v1")
+REPO = os.environ.get("BIKORET_GITEA_REPO", "heaviness510/bikoret")
+DRY = os.environ.get("BIKORET_WATCHDOG_DRY") == "1"
+THRESHOLD = 2
+
+
+def now():
+    return datetime.now(timezone.utc)
+
+
+def log(msg):
+    os.makedirs(os.path.dirname(LOG), exist_ok=True)
+    with open(LOG, "a") as f:
+        f.write(f"{now().strftime('%Y-%m-%dT%H:%M:%SZ')} {msg}\n")
+
+
+def probe():
+    """(ok, summary, failed_checks)"""
+    try:
+        with urllib.request.urlopen(urllib.request.Request(URL, headers={"user-agent": "bikoret-watchdog"}), timeout=15) as r:
+            code, body = r.status, r.read()
+    except urllib.error.HTTPError as e:
+        code, body = e.code, e.read()
+    except Exception as e:  # DNS, TLS, timeout, refused
+        return False, f"unreachable: {type(e).__name__}: {e}", ["unreachable"]
+    try:
+        d = json.loads(body)
+        checks = d.get("checks") or {}
+        failed = sorted(k for k, v in checks.items() if v != "pass")
+        overall = d.get("overall")
+        ok = code == 200 and overall == "healthy"
+        return ok, f"HTTP {code} overall={overall} failed={failed or 'none'}", failed or ([f"HTTP {code}"] if not ok else [])
+    except Exception:
+        return False, f"HTTP {code} non-JSON body ({len(body)} bytes)", [f"HTTP {code}"]
+
+
+def token():
+    """The Gitea token: ~/.config/bikoret/gitea_watchdog_token first (a token
+    the kohen issued with write:issue — the clone token in ~/.git-credentials
+    has only the code scopes and Gitea answers 403 on an issue), else that."""
+    try:
+        t = open(f"{HOME}/.config/bikoret/gitea_watchdog_token").read().strip()
+        if t:
+            return t
+    except FileNotFoundError:
+        pass
+    try:
+        for line in open(f"{HOME}/.git-credentials"):
+            m = re.match(r"https?://([^:]+):([^@]+)@", line.strip())
+            if m:
+                return m.group(2)
+    except FileNotFoundError:
+        pass
+    return None
+
+
+def gitea(method, path, payload=None):
+    if DRY:
+        log(f"DRY {method} {path} {json.dumps(payload)[:120] if payload else ''}")
+        return {"number": -1, "html_url": "(dry)"}
+    tok = token()
+    if not tok:
+        log("no Gitea token in ~/.git-credentials — cannot open an issue")
+        return None
+    data = json.dumps(payload).encode() if payload is not None else None
+    req = urllib.request.Request(f"{API}/repos/{REPO}{path}", data=data, method=method,
+                                 headers={"authorization": f"token {tok}", "content-type": "application/json", "accept": "application/json"})
+    try:
+        with urllib.request.urlopen(req, timeout=20) as r:
+            return json.loads(r.read() or b"{}")
+    except urllib.error.HTTPError as e:
+        log(f"Gitea {method} {path} -> HTTP {e.code}: {e.read()[:120]!r}")
+        return None
+    except Exception as e:
+        log(f"Gitea {method} {path} failed: {type(e).__name__}: {e}")
+        return None
+
+
+def load_state():
+    try:
+        return json.load(open(STATE))
+    except Exception:
+        return {"fails": 0, "down_since": None, "issue": None, "failed": []}
+
+
+def save_state(s):
+    os.makedirs(os.path.dirname(STATE), exist_ok=True)
+    tmp = STATE + ".tmp"
+    json.dump(s, open(tmp, "w"))
+    os.replace(tmp, STATE)
+
+
+def main():
+    s = load_state()
+    ok, summary, failed = probe()
+    t = now().strftime("%Y-%m-%d %H:%M:%SZ")
+
+    if ok:
+        if s["issue"] is not None or s["down_since"]:
+            since = s["down_since"] or t
+            mins = int((now() - datetime.fromisoformat(since.replace("Z", "+00:00").replace(" ", "T"))).total_seconds() // 60)
+            log(f"RECOVERED after {mins} min ({summary})")
+            if s["issue"] is not None:
+                gitea("POST", f"/issues/{s['issue']}/comments", {"body": f"Recovered at {t} after {mins} min. `{summary}`"})
+                gitea("PATCH", f"/issues/{s['issue']}", {"state": "closed"})
+        elif s["fails"]:
+            log(f"ok again after {s['fails']} failed probe(s), no alert raised ({summary})")
+        save_state({"fails": 0, "down_since": None, "issue": None, "failed": []})
+        return 0
+
+    s["fails"] += 1
+    if not s["down_since"]:
+        s["down_since"] = t
+    log(f"FAIL #{s['fails']}: {summary}")
+
+    if s["fails"] >= THRESHOLD and s["issue"] is None:
+        body = (f"bikoret.ch health has failed {s['fails']} consecutive probes since {s['down_since']}.\n\n"
+                f"Last probe: `{summary}`\n\nFailed checks: {', '.join(failed) or 'none named'}\n\n"
+                "What to look at first: the vav-pipeline startup log (`kubectl -n shorashim logs <pod> -c vav-pipeline | grep -v health`) — "
+                "a Doorpost drift, stale manna at the day boundary, and a missing anchor all look the same from outside. "
+                "Claude Code behind the gate is down with the site: `claude-direct` or `claude-key` (KOHEN.md, Fallback).\n\n"
+                "Opened by bikoret-watchdog on the dev VM; it will comment and close this issue on recovery.")
+        r = gitea("POST", "/issues", {"title": f"WATCHDOG: bikoret.ch down since {s['down_since']} ({', '.join(failed)[:60]})", "body": body, "labels": []})
+        if r:
+            s["issue"] = r.get("number")
+            s["failed"] = failed
+            log(f"ALERT issue #{s['issue']} opened: {r.get('html_url')}")
+    elif s["issue"] is not None and failed != s.get("failed"):
+        gitea("POST", f"/issues/{s['issue']}/comments", {"body": f"{t}: failed checks now {', '.join(failed)} — `{summary}`"})
+        s["failed"] = failed
+        log(f"issue #{s['issue']} updated: failed={failed}")
+
+    save_state(s)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())