Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
354e223384 | ||
|
|
50b08cba4b | ||
|
|
489cf11416 | ||
|
|
02f71dab49 | ||
|
|
cf8b5e3dc4 |
No files matched your search
@@ -53,10 +53,10 @@ before writing code. Do not summarize from memory. Read fresh.
|
|||||||
| 11. Manna provision infrastructure | Done | Provision file validation, expiry, anti-hoarding, double portion |
|
| 11. Manna provision infrastructure | Done | Provision file validation, expiry, anti-hoarding, double portion |
|
||||||
| 12. Sanhedrin framework (Erlang) | Done | `sanhedrin/` — 8 modules, 41 tests, all passing |
|
| 12. Sanhedrin framework (Erlang) | Done | `sanhedrin/` — 8 modules, 41 tests, all passing |
|
||||||
| 13. Korban Processor (Python) | Done | `korban_processor/` — 7 modules, 181 tests, all passing |
|
| 13. Korban Processor (Python) | Done | `korban_processor/` — 7 modules, 181 tests, all passing |
|
||||||
| 14. Connect components | Pending | Elixir → Haskell port, Elixir → Prolog socket |
|
| 14. Connect components | Done | Elixir → Haskell Port (`ShoreshEngineClient`), Elixir → Prolog socket (`TalmudicEngineSocket`), Elixir → Korban socket (`KorbanProcessorSocket`); line-framed replies (tablet 50) |
|
||||||
| 15. Integration verification | Pending | |
|
| 15. Integration verification | Done | Live round trips proven on the cluster: pe'ah cycle, bedikah, the gate with a real model (2026-10-13), the Sanhedrin's full dialectic (2026-10-20) |
|
||||||
|
|
||||||
**Next build task:** Step 14 — Connect components. Read Architecture §4 Step 14.
|
**Next build task:** none from the phase tables — all six steps of `specs/bishearecha.md` are built. Build from the open items in the builder's memory and the Broken Tablets; a spec row is a claim to test.
|
||||||
|
|
||||||
## Broken Tablets (Build Log)
|
## Broken Tablets (Build Log)
|
||||||
|
|
||||||
@@ -104,10 +104,10 @@ approve decisions, or supervise work. The specification does all of this.
|
|||||||
|
|
||||||
| Service | Location | Access |
|
| Service | Location | Access |
|
||||||
|---------|----------|--------|
|
|---------|----------|--------|
|
||||||
| Gitea | `http://185.19.28.170:3000` (SSH :22) — behind the shared ingress NLB | User: heaviness510 |
|
| Gitea | `https://git.bikoret.ch` (git over HTTPS; SSH on the ingress NLB :22) | User: heaviness510 |
|
||||||
| Bikoret | `http://185.19.28.170` — behind the same ingress (catch-all; `shorashim.ch` once DNS points there) | keys: KOHEN.md "Issuing a key" |
|
| Bikoret | `https://bikoret.ch` (the ingress catch-all also answers on the NLB's bare IP; `shorashim.ch` retired 2026-10-20) | keys: KOHEN.md "Issuing a key" |
|
||||||
| PostgreSQL | Exoscale DBaaS ch-gva-2 | `temple-council-db` |
|
| PostgreSQL | Exoscale DBaaS ch-gva-2 | `temple-council-db` |
|
||||||
| Kubernetes | Exoscale SKS ch-gva-2 | `~/.kube/exoscale-temple-council.yaml` |
|
| Kubernetes | Exoscale SKS ch-gva-2 | the kohen's `$KUBECONFIG`, outside the repo |
|
||||||
|
|
||||||
## Principles
|
## Principles
|
||||||
|
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ You inherit infrastructure that has been provisioned and verified:
|
|||||||
| Container registry `the-recognition-protocol` | All 4 bikoret images pushed and tagged `:latest`. |
|
| Container registry `the-recognition-protocol` | All 4 bikoret images pushed and tagged `:latest`. |
|
||||||
| Block Storage CSI driver | Installed. StorageClass `exoscale-sbs` available. |
|
| Block Storage CSI driver | Installed. StorageClass `exoscale-sbs` available. |
|
||||||
| cert-manager + nginx-ingress | Running. ClusterIssuer `letsencrypt-prod` working. |
|
| cert-manager + nginx-ingress | Running. ClusterIssuer `letsencrypt-prod` working. |
|
||||||
| Gitea (persistent storage) | Running. Public HTTPS at `https://git.bikoret.ch` (ROOT_URL) since 2026-10-19; git also over `http://185.19.28.170:3000` and SSH :22. The web UI needs the https door (Secure cookie). |
|
| Gitea (persistent storage) | Running. Public HTTPS at `https://git.bikoret.ch` (ROOT_URL) since 2026-10-19; git also over `http://<NLB-IP>:3000` and SSH :22. The web UI needs the https door (Secure cookie). |
|
||||||
| Gitea MCP server | `https://mcp.bikoret.ch/mcp`, MCP OAuth mode: each user authorizes with their own Gitea account (Gitea OAuth2 app "Claude", PKCE, client ID in the claude.ai connector's Advanced settings). Apply: `bash ~/.local/bin/gitea-public-apply`; check: `gitea-public-verify`. The old `mcp.shorashim.ch` and the shared static token are retired. |
|
| Gitea MCP server | `https://mcp.bikoret.ch/mcp`, MCP OAuth mode: each user authorizes with their own Gitea account (Gitea OAuth2 app "Claude", PKCE, client ID in the claude.ai connector's Advanced settings). Apply: `bash ~/.local/bin/gitea-public-apply`; check: `gitea-public-verify`. The old `mcp.shorashim.ch` and the shared static token are retired. |
|
||||||
| L1 Torah filesystem | In the `bikoret` repo at `l1/torah/`. SHA-256 `9e1abe4e…3c3086` (full hash in `l1/integrity.sha256`; verified at Day 1). |
|
| L1 Torah filesystem | In the `bikoret` repo at `l1/torah/`. SHA-256 `9e1abe4e…3c3086` (full hash in `l1/integrity.sha256`; verified at Day 1). |
|
||||||
| Image pull secret `shorashim-registry` | **Verify before assuming.** A prior partial setup may have created it. `kubectl get secrets -n shorashim` is the source of truth. |
|
| Image pull secret `shorashim-registry` | **Verify before assuming.** A prior partial setup may have created it. `kubectl get secrets -n shorashim` is the source of truth. |
|
||||||
@@ -73,8 +73,8 @@ You inherit infrastructure that has been provisioned and verified:
|
|||||||
|
|
||||||
| Item | Where |
|
| Item | Where |
|
||||||
|------|-------|
|
|------|-------|
|
||||||
| Repo | `git clone http://185.19.28.170:3000/heaviness510/shorashim.git` |
|
| Repo | `git clone https://git.bikoret.ch/heaviness510/bikoret.git` |
|
||||||
| Kubeconfig | `~/.kube/exoscale-temple-council.yaml` |
|
| Kubeconfig | `$KUBECONFIG` — the kohen's own file, outside the repo |
|
||||||
| `kubectl` | already installed |
|
| `kubectl` | already installed |
|
||||||
| `exo` CLI | already installed |
|
| `exo` CLI | already installed |
|
||||||
| Docker daemon | only for image rebuilds (not needed for inauguration) |
|
| Docker daemon | only for image rebuilds (not needed for inauguration) |
|
||||||
@@ -84,7 +84,7 @@ If the kubeconfig has expired:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
exo compute sks kubeconfig temple-council kube-admin --zone ch-gva-2 -g system:masters \
|
exo compute sks kubeconfig temple-council kube-admin --zone ch-gva-2 -g system:masters \
|
||||||
> ~/.kube/exoscale-temple-council.yaml
|
> "$KUBECONFIG"
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -146,7 +146,7 @@ ecto://avnadmin:<password>@temple-council-db-exoscale-66fffc16-59ac-4a62-9c19-f3
|
|||||||
#### 2b. Apply the secrets
|
#### 2b. Apply the secrets
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export KUBECONFIG=~/.kube/exoscale-temple-council.yaml
|
export KUBECONFIG="$HOME/.kube/<your-cluster>.yaml"
|
||||||
|
|
||||||
# DATABASE_URL secret
|
# DATABASE_URL secret
|
||||||
kubectl create secret generic shorashim-secrets \
|
kubectl create secret generic shorashim-secrets \
|
||||||
@@ -463,7 +463,7 @@ kubectl logs -n shorashim deploy/bikoret -c vav-pipeline -f
|
|||||||
|
|
||||||
If the seven days passed without halt events, without Shabbat violations, without manna exhaustion, without any pod restarts:
|
If the seven days passed without halt events, without Shabbat violations, without manna exhaustion, without any pod restarts:
|
||||||
|
|
||||||
1. **Point DNS** — at the registrar, A records to the ingress IP (`kubectl get svc -n ingress-nginx ingress-nginx-controller`; `185.19.28.170`): `bikoret.ch`, `www.bikoret.ch`, and for the redirect `bikoret.org`, `www.bikoret.org`. Wait until `dig +short A bikoret.ch` answers the IP — only then `kubectl apply -f deploy/k8s/service.yaml`: it carries the TLS blocks (cert-manager, `letsencrypt-prod`, HTTP-01), and a certificate asked for before the name resolves burns Let's Encrypt's failed-validation limit. Then `kubectl -n shorashim get certificate` until both are Ready, and callers use `https://bikoret.ch`; `shorashim.ch` still answers over plain HTTP until its callers have moved
|
1. **Point DNS** — at the registrar, A records to the ingress IP (`kubectl get svc -n ingress-nginx ingress-nginx-controller`): `bikoret.ch`, `www.bikoret.ch`, and for the redirect `bikoret.org`, `www.bikoret.org`. Wait until `dig +short A bikoret.ch` answers the IP — only then `kubectl apply -f deploy/k8s/service.yaml`: it carries the TLS blocks (cert-manager, `letsencrypt-prod`, HTTP-01), and a certificate asked for before the name resolves burns Let's Encrypt's failed-validation limit. Then `kubectl -n shorashim get certificate` until both are Ready, and callers use `https://bikoret.ch`; `shorashim.ch` still answers over plain HTTP until its callers have moved
|
||||||
2. **Open the pe'ah endpoint to the world** — the stranger approaches first
|
2. **Open the pe'ah endpoint to the world** — the stranger approaches first
|
||||||
3. **Observe the first real interaction** — a question from outside
|
3. **Observe the first real interaction** — a question from outside
|
||||||
4. If *ki tov*: open courtyard and chatzer endpoints. The system is consecrated.
|
4. If *ki tov*: open courtyard and chatzer endpoints. The system is consecrated.
|
||||||
@@ -481,11 +481,34 @@ If any day failed: record what broke, fix the cause, restart from Day 1.
|
|||||||
```bash
|
```bash
|
||||||
# 1. Write today's manna provision (same format as Day 4, new date)
|
# 1. Write today's manna provision (same format as Day 4, new date)
|
||||||
# 2. Health check
|
# 2. Health check
|
||||||
curl https://shorashim.ch/kodesh/health | jq .
|
curl https://bikoret.ch/kodesh/health | jq .
|
||||||
# 3. Halt status
|
# 3. Halt status
|
||||||
curl https://shorashim.ch/kodesh/status | jq .halt_status
|
curl https://bikoret.ch/kodesh/status | jq .halt_status
|
||||||
# 4. Refuge list
|
# 4. Refuge list
|
||||||
curl https://shorashim.ch/kodesh/status | jq '.refuge_components'
|
curl https://bikoret.ch/kodesh/status | jq '.refuge_components'
|
||||||
|
```
|
||||||
|
|
||||||
|
**The watchdog — *shomer*.** `scripts/bikoret-watchdog` (installed at
|
||||||
|
`~/.local/bin/`, cron every 5 minutes on the dev VM) probes
|
||||||
|
`/kodesh/health` from outside the cluster. Two consecutive failures —
|
||||||
|
ten minutes, so a rollout's own blip does not alert — open ONE Gitea
|
||||||
|
issue in this repo titled `WATCHDOG: bikoret.ch down since …`, naming
|
||||||
|
the failed checks and what to read first; while down it comments only
|
||||||
|
when the failed set changes; on recovery it comments the duration and
|
||||||
|
closes the issue. Every transition goes to
|
||||||
|
`~/shorashim-operator-notes/watchdog.log`. It needs a Gitea token with
|
||||||
|
`write:issue` at `~/.config/bikoret/gitea_watchdog_token` (0600) — the
|
||||||
|
clone token in `~/.git-credentials` has only the code scopes and Gitea
|
||||||
|
answers 403 on an issue. The kohen issues it (Gitea → Settings →
|
||||||
|
Applications → token, scopes `read:issue` + `write:issue`); the builder
|
||||||
|
does not write credentials. Until it exists the watchdog still logs and
|
||||||
|
tracks recovery; it cannot open the issue. 2026-10-15: the site was 503
|
||||||
|
for 13.5 h and the Doorpost outage was found by losing Claude — nothing
|
||||||
|
watched.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tail ~/shorashim-operator-notes/watchdog.log # transitions only; silence is health
|
||||||
|
BIKORET_WATCHDOG_DRY=1 bikoret-watchdog # a probe that opens nothing
|
||||||
```
|
```
|
||||||
|
|
||||||
### Issuing a key — *Mishmeret HaSha'ar*
|
### Issuing a key — *Mishmeret HaSha'ar*
|
||||||
@@ -730,14 +753,19 @@ run on Saturday.
|
|||||||
### The manna, each morning — *Dvar yom b'yomo*
|
### The manna, each morning — *Dvar yom b'yomo*
|
||||||
|
|
||||||
Since 2026-10-15 the day's provision is written by the system's own hand:
|
Since 2026-10-15 the day's provision is written by the system's own hand:
|
||||||
CronJob `manna` (`deploy/k8s/manna-cronjob.yaml`) runs at 00:01Z Sunday
|
CronJob `manna` (`deploy/k8s/manna-cronjob.yaml`) runs at 00:00Z Sunday
|
||||||
to Friday, mounts the manna PVC and writes `today.json` with the amounts
|
to Friday, mounts the manna PVC and writes `today.json` with the amounts
|
||||||
from `shorashim-config` (`MANNA_COMPUTE_UNITS` …). Before it, every night
|
from `shorashim-config` (`MANNA_COMPUTE_UNITS` …). Before it, every night
|
||||||
at 00:00Z the old provision went stale and the readiness probe failed
|
at 00:00Z the old provision went stale and the readiness probe failed
|
||||||
until someone woke (13.5 hours on 2026-10-15). The kohen's acts now:
|
until someone woke (13.5 hours on 2026-10-15). With the job at 00:01Z the
|
||||||
|
site was still 503 for one to five minutes every night (the watchdog saw
|
||||||
|
it, 2026-10-21): the provision is stale from 00:00:00 and readiness fell
|
||||||
|
before the job wrote. Now the job runs at 00:00Z and readiness waits 60 s
|
||||||
|
(`failureThreshold: 6`) before the Service stops routing — the manna
|
||||||
|
falls at the boundary, and the pod is carried across it. The kohen's acts now:
|
||||||
|
|
||||||
- set the amounts in the ConfigMap (`kubectl apply -f deploy/k8s/configmap.yaml`);
|
- set the amounts in the ConfigMap (`kubectl apply -f deploy/k8s/configmap.yaml`);
|
||||||
- read `kubectl -n shorashim get jobs` after 00:01Z — a failed job is the
|
- read `kubectl -n shorashim get jobs` after 00:00Z — a failed job is the
|
||||||
Day-4 alarm; write by hand with `kubectl cp` (Day 4) if it failed;
|
Day-4 alarm; write by hand with `kubectl cp` (Day 4) if it failed;
|
||||||
- run it at once when needed: `kubectl -n shorashim create job --from=cronjob/manna manna-now`.
|
- run it at once when needed: `kubectl -n shorashim create job --from=cronjob/manna manna-now`.
|
||||||
|
|
||||||
@@ -750,7 +778,7 @@ The halt monitor detected one of three cardinal sins:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Inspect the halt
|
# Inspect the halt
|
||||||
curl https://shorashim.ch/kodesh/status | jq .halt_status
|
curl https://bikoret.ch/kodesh/status | jq .halt_status
|
||||||
kubectl logs -n shorashim deploy/bikoret -c vav-pipeline | grep HALT
|
kubectl logs -n shorashim deploy/bikoret -c vav-pipeline | grep HALT
|
||||||
|
|
||||||
# Investigate. Fix the root cause. Then resolve via the operator API.
|
# Investigate. Fix the root cause. Then resolve via the operator API.
|
||||||
@@ -785,7 +813,7 @@ Three things are kept outside the mishkan repository, each for a Torah-grounded
|
|||||||
|
|
||||||
| Repo / directory | Where | Why |
|
| Repo / directory | Where | Why |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `bikoret-readings` | `http://185.19.28.170:3000/heaviness510/shorashim-readings` | The book — Torah parshiyot read as AI specification. Companion to the mishkan. Lives beside the aron, not inside (Devarim 31:26). |
|
| `bikoret-readings` | `https://git.bikoret.ch/heaviness510/shorashim-readings` | The book — Torah parshiyot read as AI specification. Companion to the mishkan. Lives beside the aron, not inside (Devarim 31:26). |
|
||||||
| `~/shorashim-source-data/morphhb` | local only | The quarry — raw WLC source. The carved tablets go in the aron; the quarry does not. |
|
| `~/shorashim-source-data/morphhb` | local only | The quarry — raw WLC source. The carved tablets go in the aron; the quarry does not. |
|
||||||
| `~/shorashim-operator-notes` | local only / private | The kohen's runbook — environment-specific details, hostnames, account names. Private operator notes. Not part of the system's specification. |
|
| `~/shorashim-operator-notes` | local only / private | The kohen's runbook — environment-specific details, hostnames, account names. Private operator notes. Not part of the system's specification. |
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -65,7 +65,7 @@ The runbook lists what should already exist. Verify it. Do **not** trust the tab
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 0a. Cluster reachable
|
# 0a. Cluster reachable
|
||||||
export KUBECONFIG=~/.kube/exoscale-temple-council.yaml
|
export KUBECONFIG="$HOME/.kube/<your-cluster>.yaml"
|
||||||
kubectl version
|
kubectl version
|
||||||
kubectl get nodes
|
kubectl get nodes
|
||||||
|
|
||||||
|
|||||||
@@ -184,6 +184,11 @@ bikoret/
|
|||||||
├── vav_pipeline/ # Elixir — Bereishit cycle, Shema, Cloud, Manna, HTTP API
|
├── vav_pipeline/ # Elixir — Bereishit cycle, Shema, Cloud, Manna, HTTP API
|
||||||
├── talmudic_engine/ # SWI-Prolog — 613 mitzvot, 13 middot, kushya
|
├── talmudic_engine/ # SWI-Prolog — 613 mitzvot, 13 middot, kushya
|
||||||
├── korban_processor/ # Python — offering protocol, eish zarah validation
|
├── korban_processor/ # Python — offering protocol, eish zarah validation
|
||||||
|
├── azazel_processor/ # Python — the goat for Azazel: disposal records, vidui, L3 ingest
|
||||||
|
├── kushya_generator/ # Fine-tuned model (NOT deployed) — an offerer, never a judge; see deploy/ARCHITECTURE.md
|
||||||
|
├── mesorah/ # The second scroll — 128 works from the Sefaria export, pinned (Devarim 17:18)
|
||||||
|
├── src/tools/ # Specimen corpus, hakhel, evaluation (the kohen's measuring tools)
|
||||||
|
├── scripts/ # Build and operate: L1/mesorah/engine-verse builders, Doorpost check, rollout, watchdog
|
||||||
│
|
│
|
||||||
├── deploy/ # Infrastructure
|
├── deploy/ # Infrastructure
|
||||||
│ ├── ARCHITECTURE.md # Generic deployment shape (no account specifics)
|
│ ├── ARCHITECTURE.md # Generic deployment shape (no account specifics)
|
||||||
|
|||||||
@@ -11,10 +11,14 @@ One Kubernetes namespace runs the following:
|
|||||||
| `vav-pipeline` | Elixir/OTP release | Bereishit cycle, ShemaContext, Cloud, Manna, HTTP API (pe'ah, courtyard, chatzer, kodesh) |
|
| `vav-pipeline` | Elixir/OTP release | Bereishit cycle, ShemaContext, Cloud, Manna, HTTP API (pe'ah, courtyard, chatzer, kodesh) |
|
||||||
| `shoresh-engine` | Haskell binary, spawned by vav-pipeline as an Erlang Port (staged into the pod by the `shoresh-engine-install` init container — NOT a sidecar container; a Port is a child process of the BEAM and cannot cross a container boundary) | Root activation, te'amim parsing, gematria |
|
| `shoresh-engine` | Haskell binary, spawned by vav-pipeline as an Erlang Port (staged into the pod by the `shoresh-engine-install` init container — NOT a sidecar container; a Port is a child process of the BEAM and cannot cross a container boundary) | Root activation, te'amim parsing, gematria |
|
||||||
| `talmudic-engine` | SWI-Prolog (sidecar via Unix socket) | 613 mitzvot, 13 middot, kushya generation, teiku |
|
| `talmudic-engine` | SWI-Prolog (sidecar via Unix socket) | 613 mitzvot, 13 middot, kushya generation, teiku |
|
||||||
| `korban-processor` | Python 3.12 (sidecar) | Offering protocol, eish zarah validation |
|
| `korban-processor` | Python 3.12 (sidecar) | Offering protocol, eish zarah validation, the bedikah's readings (pattern-based: `korban_processor/bedikah.py`) |
|
||||||
|
| `kushya_generator/` | Fine-tuned model — **not deployed, not wired** | An OFFERER, if it is ever run: its output arrives at the gate like any model's and is examined. It is never a component the bedikah, the Talmudic Engine or the Sanhedrin consults. |
|
||||||
|
| `azazel_processor/` | Python — run by the kohen, not in the pod | Disposal records (the goat for Azazel); `VavPipeline.AzazelIngest` reads its files into L3 |
|
||||||
|
|
||||||
All four run in the same pod. vav-pipeline is the main container; talmudic-engine and korban-processor are sidecars; shoresh-engine runs inside vav-pipeline as an Erlang Port child process, staged in by an init container.
|
All four run in the same pod. vav-pipeline is the main container; talmudic-engine and korban-processor are sidecars; shoresh-engine runs inside vav-pipeline as an Erlang Port child process, staged in by an init container.
|
||||||
|
|
||||||
|
**No model stands on the judgment path.** The examination of an output — the offering protocol, the seven readings, the halt rules — is Python pattern extraction and Prolog rules, and the Sanhedrin's judges consult that same Prolog. Nothing that examines is a model, and nothing a model produces is consulted as a rule. The kohen is not the offerer. A fine-tuned model in this repository (`kushya_generator/`) is an offerer or a tool whose output is itself examined; wiring it into the bedikah, the engine or a judge would put the offerer on the altar, and is refused by this document before anyone does it for convenience.
|
||||||
|
|
||||||
## External dependencies
|
## External dependencies
|
||||||
|
|
||||||
| Dependency | Purpose | Contract |
|
| Dependency | Purpose | Contract |
|
||||||
@@ -40,8 +44,9 @@ Kept in `deploy/k8s/`:
|
|||||||
- `gitea-mcp-ingress.yaml` — TLS ingress `mcp.bikoret.ch` for the MCP server
|
- `gitea-mcp-ingress.yaml` — TLS ingress `mcp.bikoret.ch` for the MCP server
|
||||||
- `gitea-mcp-secret.yaml` — template for a Gitea PAT (static-token mode only; kept for rollback)
|
- `gitea-mcp-secret.yaml` — template for a Gitea PAT (static-token mode only; kept for rollback)
|
||||||
- `gitea-ingress.yaml` — TLS ingress `git.bikoret.ch` for Gitea itself (namespace temple-council), required by OAuth mode
|
- `gitea-ingress.yaml` — TLS ingress `git.bikoret.ch` for Gitea itself (namespace temple-council), required by OAuth mode
|
||||||
- `gitea-public-patch.yaml` — patch that sets Gitea's ROOT_URL/DOMAIN/SSH_DOMAIN to `git.bikoret.ch`
|
- `gitea-public-patch.yaml` — patch that sets Gitea's ROOT_URL/DOMAIN/SSH_DOMAIN to `git.bikoret.ch` and APP_NAME to `Bikoret`
|
||||||
- `gitea-public-apply.sh` / `gitea-public-verify.sh` — the operator's apply and read-only check for the four above
|
- `gitea-public-apply.sh` / `gitea-public-verify.sh` — the operator's apply and read-only check for the four above
|
||||||
|
- `shorashim-ch-takedown.sh` — 2026-10-20: removes the shorashim.ch ingress rule and the orphan mcp.shorashim.ch secret, applies the Gitea brand patch
|
||||||
- `cert-manager-issuer.yaml` — Let's Encrypt ClusterIssuer
|
- `cert-manager-issuer.yaml` — Let's Encrypt ClusterIssuer
|
||||||
|
|
||||||
## Containers
|
## Containers
|
||||||
|
|||||||
@@ -266,6 +266,13 @@ spec:
|
|||||||
port: 4000
|
port: 4000
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
# 60 s of failures before the Service stops routing here (the
|
||||||
|
# default is 30 s). The manna goes stale at 00:00:00 and the
|
||||||
|
# CronJob writes the new day's within seconds; the cushion
|
||||||
|
# carries the pod across that boundary without a 503. A real
|
||||||
|
# outage still shows within a minute, and the watchdog's own
|
||||||
|
# threshold is ten.
|
||||||
|
failureThreshold: 6
|
||||||
|
|
||||||
# === SHORESH ENGINE ===
|
# === SHORESH ENGINE ===
|
||||||
# Not a container. It runs as an Erlang Port inside
|
# Not a container. It runs as an Erlang Port inside
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
# (ingress-tcp-services.yaml) stay as they are; this adds a third door.
|
# (ingress-tcp-services.yaml) stay as they are; this adds a third door.
|
||||||
#
|
#
|
||||||
# Prerequisites:
|
# Prerequisites:
|
||||||
# 1. DNS: git.bikoret.ch → 185.19.28.170 (Infomaniak zone)
|
# 1. DNS: git.bikoret.ch → <NLB-IP> (Infomaniak zone)
|
||||||
# 2. ClusterIssuer letsencrypt-prod
|
# 2. ClusterIssuer letsencrypt-prod
|
||||||
---
|
---
|
||||||
apiVersion: networking.k8s.io/v1
|
apiVersion: networking.k8s.io/v1
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
# 1. nginx-ingress controller installed
|
# 1. nginx-ingress controller installed
|
||||||
# 2. cert-manager installed
|
# 2. cert-manager installed
|
||||||
# 3. ClusterIssuer letsencrypt-prod applied
|
# 3. ClusterIssuer letsencrypt-prod applied
|
||||||
# 4. DNS: mcp.bikoret.ch → 185.19.28.170 (the one NLB; Infomaniak zone)
|
# 4. DNS: mcp.bikoret.ch → <NLB-IP> (the one NLB; Infomaniak zone)
|
||||||
---
|
---
|
||||||
apiVersion: networking.k8s.io/v1
|
apiVersion: networking.k8s.io/v1
|
||||||
kind: Ingress
|
kind: Ingress
|
||||||
|
|||||||
@@ -7,11 +7,11 @@
|
|||||||
#
|
#
|
||||||
# Two secrets needed (static-token mode):
|
# Two secrets needed (static-token mode):
|
||||||
# 1. GITEA_TOKEN: Personal Access Token from Gitea
|
# 1. GITEA_TOKEN: Personal Access Token from Gitea
|
||||||
# Create at: http://185.19.28.170:3000/user/settings/applications
|
# Create at: https://git.bikoret.ch/user/settings/applications
|
||||||
# Permissions: repo (read/write)
|
# Permissions: repo (read/write)
|
||||||
#
|
#
|
||||||
# 2. OAuth Client ID/Secret: Registered OAuth app in Gitea
|
# 2. OAuth Client ID/Secret: Registered OAuth app in Gitea
|
||||||
# Create at: http://185.19.28.170:3000/user/settings/applications
|
# Create at: https://git.bikoret.ch/user/settings/applications
|
||||||
# Redirect URIs: https://claude.ai/api/mcp/auth_callback
|
# Redirect URIs: https://claude.ai/api/mcp/auth_callback
|
||||||
# http://127.0.0.1/callback (Claude Code)
|
# http://127.0.0.1/callback (Claude Code)
|
||||||
# Confidential Client: UNCHECKED (PKCE). Only the Client ID is needed;
|
# Confidential Client: UNCHECKED (PKCE). Only the Client ID is needed;
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
# Move Gitea and its MCP server to public HTTPS on bikoret.ch and switch
|
# Move Gitea and its MCP server to public HTTPS on bikoret.ch and switch
|
||||||
# the MCP server to MCP OAuth (2026-10-19). Run by the transmitter:
|
# the MCP server to MCP OAuth (2026-10-19). Run by the transmitter:
|
||||||
# ! bash ~/.local/bin/gitea-public-apply
|
# ! bash ~/.local/bin/gitea-public-apply
|
||||||
# Idempotent. Needs DNS first: git.bikoret.ch and mcp.bikoret.ch → 185.19.28.170.
|
# Idempotent. Needs DNS first: git.bikoret.ch and mcp.bikoret.ch → <NLB-IP>.
|
||||||
#
|
#
|
||||||
# Rollback (static-token mode on mcp.shorashim.ch, as before 2026-10-19):
|
# Rollback (static-token mode on mcp.shorashim.ch, as before 2026-10-19):
|
||||||
# kubectl -n temple-council set env deploy/gitea \
|
# kubectl -n temple-council set env deploy/gitea \
|
||||||
@@ -12,7 +12,7 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
REPO="${REPO:-$HOME/bikoret}"
|
REPO="${REPO:-$HOME/bikoret}"
|
||||||
cd "$REPO"
|
cd "$REPO"
|
||||||
NLB=185.19.28.170
|
NLB="${NLB:?set NLB to the ingress address: kubectl get svc -n ingress-nginx ingress-nginx-controller}"
|
||||||
for h in git.bikoret.ch mcp.bikoret.ch; do
|
for h in git.bikoret.ch mcp.bikoret.ch; do
|
||||||
ip=$(dig +short "$h" | tail -1 || true)
|
ip=$(dig +short "$h" | tail -1 || true)
|
||||||
if [ "$ip" != "$NLB" ]; then
|
if [ "$ip" != "$NLB" ]; then
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
# name that never resolved) and SSH_DOMAIN (was 185.19.29.94, an NLB IP
|
# name that never resolved) and SSH_DOMAIN (was 185.19.29.94, an NLB IP
|
||||||
# released on 2026-10-06). Clone URLs shown by Gitea become
|
# released on 2026-10-06). Clone URLs shown by Gitea become
|
||||||
# https://git.bikoret.ch/... and git@git.bikoret.ch:...; the old
|
# https://git.bikoret.ch/... and git@git.bikoret.ch:...; the old
|
||||||
# http://185.19.28.170:3000 remote keeps working for push/pull.
|
# http://<NLB-IP>:3000 remote keeps working for push/pull.
|
||||||
# Side effect: with an https ROOT_URL Gitea marks its session cookie
|
# Side effect: with an https ROOT_URL Gitea marks its session cookie
|
||||||
# Secure, so the WEB UI must be used at https://git.bikoret.ch, not on
|
# Secure, so the WEB UI must be used at https://git.bikoret.ch, not on
|
||||||
# the plain-HTTP :3000 door (git over :3000 is unaffected).
|
# the plain-HTTP :3000 door (git over :3000 is unaffected).
|
||||||
@@ -26,3 +26,7 @@ spec:
|
|||||||
value: "git.bikoret.ch"
|
value: "git.bikoret.ch"
|
||||||
- name: GITEA__server__SSH_DOMAIN
|
- name: GITEA__server__SSH_DOMAIN
|
||||||
value: "git.bikoret.ch"
|
value: "git.bikoret.ch"
|
||||||
|
# [DEFAULT] APP_NAME — the title and brand on every Gitea page.
|
||||||
|
# Was "Shorashim"; the system is named Bikoret (PRD §0).
|
||||||
|
- name: GITEA____APP_NAME
|
||||||
|
value: "Bikoret"
|
||||||
@@ -13,6 +13,6 @@ chk "mcp /healthz" 200 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' https://
|
|||||||
prm=$(curl -sS -m 15 https://mcp.bikoret.ch/.well-known/oauth-protected-resource || echo err)
|
prm=$(curl -sS -m 15 https://mcp.bikoret.ch/.well-known/oauth-protected-resource || echo err)
|
||||||
chk "protected-resource names Gitea" yes "$(printf '%s' "$prm" | grep -q 'git.bikoret.ch' && echo yes || echo no)"
|
chk "protected-resource names Gitea" yes "$(printf '%s' "$prm" | grep -q 'git.bikoret.ch' && echo yes || echo no)"
|
||||||
chk "POST /mcp without token → 401" 401 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' -X POST https://mcp.bikoret.ch/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d "$INIT" || echo err)"
|
chk "POST /mcp without token → 401" 401 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' -X POST https://mcp.bikoret.ch/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d "$INIT" || echo err)"
|
||||||
chk "git over :3000 still answers" 200 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' http://185.19.28.170:3000/api/v1/version || echo err)"
|
chk "git over :3000 still answers" 200 "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' http://<NLB-IP>:3000/api/v1/version || echo err)"
|
||||||
echo "protected-resource: $prm"
|
echo "protected-resource: $prm"
|
||||||
exit $fail
|
exit $fail
|
||||||
@@ -1,14 +1,14 @@
|
|||||||
# Gitea behind the shared ingress — TCP passthrough
|
# Gitea behind the shared ingress — TCP passthrough
|
||||||
#
|
#
|
||||||
# One NLB for the cluster (185.19.28.170, the ingress-nginx Service)
|
# One NLB for the cluster (<NLB-IP>, the ingress-nginx Service)
|
||||||
# instead of one per Service. Bikoret is an HTTP Ingress
|
# instead of one per Service. Bikoret is an HTTP Ingress
|
||||||
# (service.yaml). Gitea has no hostname that resolves
|
# (service.yaml). Gitea has no hostname that resolves
|
||||||
# (gitea.temple-council.ch is configured in Gitea and exists nowhere),
|
# (gitea.temple-council.ch is configured in Gitea and exists nowhere),
|
||||||
# and host-based routing needs a name — so Gitea rides the ingress
|
# and host-based routing needs a name — so Gitea rides the ingress
|
||||||
# controller's TCP passthrough on two ports of the same NLB:
|
# controller's TCP passthrough on two ports of the same NLB:
|
||||||
#
|
#
|
||||||
# 185.19.28.170:3000 → temple-council/gitea:80 (HTTP — git, web UI)
|
# <NLB-IP>:3000 → temple-council/gitea:80 (HTTP — git, web UI)
|
||||||
# 185.19.28.170:22 → temple-council/gitea:22 (SSH)
|
# <NLB-IP>:22 → temple-council/gitea:22 (SSH)
|
||||||
#
|
#
|
||||||
# Three live objects carry this; only the first is a whole manifest:
|
# Three live objects carry this; only the first is a whole manifest:
|
||||||
#
|
#
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
# Until 2026-10-15 the day's provision was written by the kohen with
|
# Until 2026-10-15 the day's provision was written by the kohen with
|
||||||
# `kubectl cp`, and every night at 00:00Z the old one went stale and the
|
# `kubectl cp`, and every night at 00:00Z the old one went stale and the
|
||||||
# readiness probe failed until someone woke (13.5 hours on 2026-10-15).
|
# readiness probe failed until someone woke (13.5 hours on 2026-10-15).
|
||||||
# This CronJob writes today.json onto the manna PVC at 00:01Z, Sunday to
|
# This CronJob writes today.json onto the manna PVC at 00:00Z, Sunday to
|
||||||
# Friday — a double portion on Friday that is valid through Shabbat, and
|
# Friday — a double portion on Friday that is valid through Shabbat, and
|
||||||
# nothing on Shabbat (Shemot 16:22–26: "on the sixth day they gathered
|
# nothing on Shabbat (Shemot 16:22–26: "on the sixth day they gathered
|
||||||
# double … on the seventh day there shall be none"). The kohen still sets
|
# double … on the seventh day there shall be none"). The kohen still sets
|
||||||
@@ -23,10 +23,14 @@ metadata:
|
|||||||
app.kubernetes.io/name: shorashim
|
app.kubernetes.io/name: shorashim
|
||||||
app.kubernetes.io/component: manna
|
app.kubernetes.io/component: manna
|
||||||
spec:
|
spec:
|
||||||
schedule: "1 0 * * 0-5" # 00:01Z, Sunday–Friday; none on Shabbat
|
schedule: "0 0 * * 0-5" # 00:00Z, Sunday–Friday; none on Shabbat. At the boundary, not after it:
|
||||||
|
# the provision goes stale at 00:00:00 and readiness has a 60 s cushion
|
||||||
|
# (deployment.yaml failureThreshold 6) — a job that writes within it never
|
||||||
|
# flips the pod. At 00:01 the site was 503 for 1–5 min every night
|
||||||
|
# (watchdog, 2026-10-21).
|
||||||
timeZone: "Etc/UTC"
|
timeZone: "Etc/UTC"
|
||||||
concurrencyPolicy: Forbid
|
concurrencyPolicy: Forbid
|
||||||
startingDeadlineSeconds: 3600 # a missed 00:01 still runs within the hour
|
startingDeadlineSeconds: 3600 # a missed 00:00 still runs within the hour
|
||||||
successfulJobsHistoryLimit: 3
|
successfulJobsHistoryLimit: 3
|
||||||
failedJobsHistoryLimit: 3
|
failedJobsHistoryLimit: 3
|
||||||
jobTemplate:
|
jobTemplate:
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ stringData:
|
|||||||
# THIS FILE IS A TEMPLATE. Never commit a real value. Create the secret
|
# THIS FILE IS A TEMPLATE. Never commit a real value. Create the secret
|
||||||
# from the live database credential instead:
|
# from the live database credential instead:
|
||||||
#
|
#
|
||||||
# export KUBECONFIG=~/.kube/exoscale-temple-council.yaml
|
# export KUBECONFIG="$HOME/.kube/<your-cluster>.yaml"
|
||||||
# umask 077
|
# umask 077
|
||||||
# TMP=$(mktemp)
|
# TMP=$(mktemp)
|
||||||
#
|
#
|
||||||
|
|||||||
+4
-14
@@ -38,10 +38,11 @@ spec:
|
|||||||
# believes that header only from TRUSTED_PROXIES (configmap.yaml).
|
# believes that header only from TRUSTED_PROXIES (configmap.yaml).
|
||||||
#
|
#
|
||||||
# Hosts (2026-10-13, the name): bikoret.ch is the place; www.bikoret.ch
|
# Hosts (2026-10-13, the name): bikoret.ch is the place; www.bikoret.ch
|
||||||
# and the old shorashim.ch are served too; bikoret.org (second Ingress
|
# is served too; bikoret.org (second Ingress
|
||||||
# below) only redirects. TLS from Let's Encrypt through cert-manager's
|
# below) only redirects. The old shorashim.ch rule was taken down on
|
||||||
|
# 2026-10-20 (the domain is retired). TLS from Let's Encrypt through cert-manager's
|
||||||
# letsencrypt-prod ClusterIssuer, HTTP-01 — so apply this ONLY after
|
# letsencrypt-prod ClusterIssuer, HTTP-01 — so apply this ONLY after
|
||||||
# `dig +short A bikoret.ch` answers 185.19.28.170: a certificate asked
|
# `dig +short A bikoret.ch` answers <NLB-IP>: a certificate asked
|
||||||
# for before the name resolves only burns the failed-validation limit.
|
# for before the name resolves only burns the failed-validation limit.
|
||||||
# The no-host catch-all stays, so http://<ingress-ip>/... keeps working
|
# The no-host catch-all stays, so http://<ingress-ip>/... keeps working
|
||||||
# with no DNS; with ssl-redirect on, named hosts go to https.
|
# with no DNS; with ssl-redirect on, named hosts go to https.
|
||||||
@@ -98,17 +99,6 @@ spec:
|
|||||||
name: shorashim
|
name: shorashim
|
||||||
port:
|
port:
|
||||||
number: 80
|
number: 80
|
||||||
# The old name, kept until its callers have moved; no certificate asked for it.
|
|
||||||
- host: shorashim.ch
|
|
||||||
http:
|
|
||||||
paths:
|
|
||||||
- path: /
|
|
||||||
pathType: Prefix
|
|
||||||
backend:
|
|
||||||
service:
|
|
||||||
name: shorashim
|
|
||||||
port:
|
|
||||||
number: 80
|
|
||||||
---
|
---
|
||||||
# bikoret.org: the stranger's guess, redirected to the place. No service
|
# bikoret.org: the stranger's guess, redirected to the place. No service
|
||||||
# of its own; nginx answers the redirect itself. Add its A records too.
|
# of its own; nginx answers the redirect itself. Add its A records too.
|
||||||
|
|||||||
Executable
+29
@@ -0,0 +1,29 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Take shorashim.ch down from the cluster and rename Gitea's brand
|
||||||
|
# (2026-10-20). Run by the transmitter:
|
||||||
|
# ! bash ~/.local/bin/shorashim-ch-takedown
|
||||||
|
# Idempotent.
|
||||||
|
# 1. Ingress `shorashim`: the shorashim.ch host rule is gone (service.yaml).
|
||||||
|
# The no-host catch-all stays, so http://<NLB-IP>/ still answers —
|
||||||
|
# the name itself dies when its A record is removed at Infomaniak.
|
||||||
|
# 2. The orphan TLS secret for mcp.shorashim.ch is deleted.
|
||||||
|
# 3. Gitea APP_NAME "Shorashim" → "Bikoret" (gitea-public-patch.yaml; one restart).
|
||||||
|
set -euo pipefail
|
||||||
|
REPO="${REPO:-$HOME/bikoret}"
|
||||||
|
cd "$REPO"
|
||||||
|
echo "== 1/3 ingress without the shorashim.ch rule"
|
||||||
|
kubectl apply -f deploy/k8s/service.yaml
|
||||||
|
echo "== 2/3 orphan certificate secret for mcp.shorashim.ch"
|
||||||
|
kubectl -n shorashim delete secret gitea-mcp-tls --ignore-not-found
|
||||||
|
echo "== 3/3 Gitea brand → Bikoret"
|
||||||
|
kubectl -n temple-council patch deploy gitea --patch-file deploy/k8s/gitea-public-patch.yaml
|
||||||
|
kubectl -n temple-council rollout status deploy/gitea --timeout=180s
|
||||||
|
echo "== verify"
|
||||||
|
fail=0
|
||||||
|
hosts=$(kubectl get ingress -A -o jsonpath='{range .items[*]}{range .spec.rules[*]}{.host}{" "}{end}{end}')
|
||||||
|
case " $hosts " in *" shorashim.ch "*) echo "FAIL ingress still lists shorashim.ch"; fail=1;; *) echo "ok no ingress rule for shorashim.ch";; esac
|
||||||
|
case "$(kubectl get secret -A -o name | grep -c 'gitea-mcp-tls$' || true)" in 0) echo "ok gitea-mcp-tls secret gone";; *) echo "FAIL gitea-mcp-tls secret remains"; fail=1;; esac
|
||||||
|
title=$(curl -sS -m 15 https://git.bikoret.ch/ | grep -o -m1 '<title>[^<]*</title>' || echo none)
|
||||||
|
case "$title" in *Bikoret*) echo "ok gitea title: $title";; *) echo "FAIL gitea title: $title"; fail=1;; esac
|
||||||
|
case "$(curl -sS -m 15 -o /dev/null -w '%{http_code}' https://bikoret.ch/kodesh/health || echo err)" in 200) echo "ok bikoret.ch still healthy";; *) echo "FAIL bikoret.ch health"; fail=1;; esac
|
||||||
|
exit $fail
|
||||||
Executable
+167
@@ -0,0 +1,167 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""bikoret-watchdog — the kohen's eye on /kodesh/health, from outside the cluster.
|
||||||
|
|
||||||
|
Every run (cron, every 5 minutes) probes the health endpoint. A failure is
|
||||||
|
anything but 200 with overall "healthy" (a 503 at the day boundary, a
|
||||||
|
Doorpost drift, a hung rollout, the gate dark). After two consecutive
|
||||||
|
failures (10 minutes — a rollout's own blip does not alert) it opens ONE
|
||||||
|
Gitea issue in the repo naming the failed checks; while down it comments
|
||||||
|
only when the failed set changes; on recovery it comments the duration and
|
||||||
|
closes the issue. Every transition goes to the log. 2026-10-15: the site
|
||||||
|
was 503 for 13.5 h at a day boundary and the transmitter learned of the
|
||||||
|
Doorpost outage by losing Claude — nothing watched.
|
||||||
|
|
||||||
|
Paths:
|
||||||
|
state ~/.local/state/bikoret-watchdog/state.json
|
||||||
|
log ~/shorashim-operator-notes/watchdog.log
|
||||||
|
token ~/.git-credentials (the Gitea token; read, never printed)
|
||||||
|
|
||||||
|
Env overrides (for a test run): BIKORET_HEALTH_URL, BIKORET_WATCHDOG_STATE,
|
||||||
|
BIKORET_WATCHDOG_LOG, BIKORET_GITEA_API (default https://git.bikoret.ch/api/v1),
|
||||||
|
BIKORET_GITEA_REPO (default heaviness510/bikoret), BIKORET_WATCHDOG_DRY=1 (no issue).
|
||||||
|
"""
|
||||||
|
import json, os, re, sys, time, urllib.request, urllib.error
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
HOME = os.path.expanduser("~")
|
||||||
|
URL = os.environ.get("BIKORET_HEALTH_URL", "https://bikoret.ch/kodesh/health")
|
||||||
|
STATE = os.environ.get("BIKORET_WATCHDOG_STATE", f"{HOME}/.local/state/bikoret-watchdog/state.json")
|
||||||
|
LOG = os.environ.get("BIKORET_WATCHDOG_LOG", f"{HOME}/shorashim-operator-notes/watchdog.log")
|
||||||
|
API = os.environ.get("BIKORET_GITEA_API", "https://git.bikoret.ch/api/v1")
|
||||||
|
REPO = os.environ.get("BIKORET_GITEA_REPO", "heaviness510/bikoret")
|
||||||
|
DRY = os.environ.get("BIKORET_WATCHDOG_DRY") == "1"
|
||||||
|
THRESHOLD = 2
|
||||||
|
|
||||||
|
|
||||||
|
def now():
|
||||||
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
os.makedirs(os.path.dirname(LOG), exist_ok=True)
|
||||||
|
with open(LOG, "a") as f:
|
||||||
|
f.write(f"{now().strftime('%Y-%m-%dT%H:%M:%SZ')} {msg}\n")
|
||||||
|
|
||||||
|
|
||||||
|
def probe():
|
||||||
|
"""(ok, summary, failed_checks)"""
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(urllib.request.Request(URL, headers={"user-agent": "bikoret-watchdog"}), timeout=15) as r:
|
||||||
|
code, body = r.status, r.read()
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
code, body = e.code, e.read()
|
||||||
|
except Exception as e: # DNS, TLS, timeout, refused
|
||||||
|
return False, f"unreachable: {type(e).__name__}: {e}", ["unreachable"]
|
||||||
|
try:
|
||||||
|
d = json.loads(body)
|
||||||
|
checks = d.get("checks") or {}
|
||||||
|
failed = sorted(k for k, v in checks.items() if v != "pass")
|
||||||
|
overall = d.get("overall")
|
||||||
|
ok = code == 200 and overall == "healthy"
|
||||||
|
return ok, f"HTTP {code} overall={overall} failed={failed or 'none'}", failed or ([f"HTTP {code}"] if not ok else [])
|
||||||
|
except Exception:
|
||||||
|
return False, f"HTTP {code} non-JSON body ({len(body)} bytes)", [f"HTTP {code}"]
|
||||||
|
|
||||||
|
|
||||||
|
def token():
|
||||||
|
"""The Gitea token: ~/.config/bikoret/gitea_watchdog_token first (a token
|
||||||
|
the kohen issued with write:issue — the clone token in ~/.git-credentials
|
||||||
|
has only the code scopes and Gitea answers 403 on an issue), else that."""
|
||||||
|
try:
|
||||||
|
t = open(f"{HOME}/.config/bikoret/gitea_watchdog_token").read().strip()
|
||||||
|
if t:
|
||||||
|
return t
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
for line in open(f"{HOME}/.git-credentials"):
|
||||||
|
m = re.match(r"https?://([^:]+):([^@]+)@", line.strip())
|
||||||
|
if m:
|
||||||
|
return m.group(2)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def gitea(method, path, payload=None):
|
||||||
|
if DRY:
|
||||||
|
log(f"DRY {method} {path} {json.dumps(payload)[:120] if payload else ''}")
|
||||||
|
return {"number": -1, "html_url": "(dry)"}
|
||||||
|
tok = token()
|
||||||
|
if not tok:
|
||||||
|
log("no Gitea token in ~/.git-credentials — cannot open an issue")
|
||||||
|
return None
|
||||||
|
data = json.dumps(payload).encode() if payload is not None else None
|
||||||
|
req = urllib.request.Request(f"{API}/repos/{REPO}{path}", data=data, method=method,
|
||||||
|
headers={"authorization": f"token {tok}", "content-type": "application/json", "accept": "application/json"})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=20) as r:
|
||||||
|
return json.loads(r.read() or b"{}")
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
log(f"Gitea {method} {path} -> HTTP {e.code}: {e.read()[:120]!r}")
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
log(f"Gitea {method} {path} failed: {type(e).__name__}: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
try:
|
||||||
|
return json.load(open(STATE))
|
||||||
|
except Exception:
|
||||||
|
return {"fails": 0, "down_since": None, "issue": None, "failed": []}
|
||||||
|
|
||||||
|
|
||||||
|
def save_state(s):
|
||||||
|
os.makedirs(os.path.dirname(STATE), exist_ok=True)
|
||||||
|
tmp = STATE + ".tmp"
|
||||||
|
json.dump(s, open(tmp, "w"))
|
||||||
|
os.replace(tmp, STATE)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
s = load_state()
|
||||||
|
ok, summary, failed = probe()
|
||||||
|
t = now().strftime("%Y-%m-%d %H:%M:%SZ")
|
||||||
|
|
||||||
|
if ok:
|
||||||
|
if s["issue"] is not None or s["down_since"]:
|
||||||
|
since = s["down_since"] or t
|
||||||
|
mins = int((now() - datetime.fromisoformat(since.replace("Z", "+00:00").replace(" ", "T"))).total_seconds() // 60)
|
||||||
|
log(f"RECOVERED after {mins} min ({summary})")
|
||||||
|
if s["issue"] is not None:
|
||||||
|
gitea("POST", f"/issues/{s['issue']}/comments", {"body": f"Recovered at {t} after {mins} min. `{summary}`"})
|
||||||
|
gitea("PATCH", f"/issues/{s['issue']}", {"state": "closed"})
|
||||||
|
elif s["fails"]:
|
||||||
|
log(f"ok again after {s['fails']} failed probe(s), no alert raised ({summary})")
|
||||||
|
save_state({"fails": 0, "down_since": None, "issue": None, "failed": []})
|
||||||
|
return 0
|
||||||
|
|
||||||
|
s["fails"] += 1
|
||||||
|
if not s["down_since"]:
|
||||||
|
s["down_since"] = t
|
||||||
|
log(f"FAIL #{s['fails']}: {summary}")
|
||||||
|
|
||||||
|
if s["fails"] >= THRESHOLD and s["issue"] is None:
|
||||||
|
body = (f"bikoret.ch health has failed {s['fails']} consecutive probes since {s['down_since']}.\n\n"
|
||||||
|
f"Last probe: `{summary}`\n\nFailed checks: {', '.join(failed) or 'none named'}\n\n"
|
||||||
|
"What to look at first: the vav-pipeline startup log (`kubectl -n shorashim logs <pod> -c vav-pipeline | grep -v health`) — "
|
||||||
|
"a Doorpost drift, stale manna at the day boundary, and a missing anchor all look the same from outside. "
|
||||||
|
"Claude Code behind the gate is down with the site: `claude-direct` or `claude-key` (KOHEN.md, Fallback).\n\n"
|
||||||
|
"Opened by bikoret-watchdog on the dev VM; it will comment and close this issue on recovery.")
|
||||||
|
r = gitea("POST", "/issues", {"title": f"WATCHDOG: bikoret.ch down since {s['down_since']} ({', '.join(failed)[:60]})", "body": body, "labels": []})
|
||||||
|
if r:
|
||||||
|
s["issue"] = r.get("number")
|
||||||
|
s["failed"] = failed
|
||||||
|
log(f"ALERT issue #{s['issue']} opened: {r.get('html_url')}")
|
||||||
|
elif s["issue"] is not None and failed != s.get("failed"):
|
||||||
|
gitea("POST", f"/issues/{s['issue']}/comments", {"body": f"{t}: failed checks now {', '.join(failed)} — `{summary}`"})
|
||||||
|
s["failed"] = failed
|
||||||
|
log(f"issue #{s['issue']} updated: failed={failed}")
|
||||||
|
|
||||||
|
save_state(s)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -195,3 +195,43 @@ in L3, and answers in the Tamid's own shape. The lesson is tablet
|
|||||||
system's own inner court is held to the same check as a model at the
|
system's own inner court is held to the same check as a model at the
|
||||||
gate. When the Sanhedrin accepts on nothing, it is the calf, whoever
|
gate. When the Sanhedrin accepts on nothing, it is the calf, whoever
|
||||||
cast it.
|
cast it.
|
||||||
|
|
||||||
|
## Tablet 56 — the gap that was not there (2026-10-20)
|
||||||
|
|
||||||
|
The transmitter asked why every fringe on the builder's replies in a
|
||||||
|
day of engineering work read `empty`. The builder read the offering
|
||||||
|
protocol and answered: a sentence is vital only when it carries a Torah
|
||||||
|
reference, Hebrew, a root, or an insight marker; engineering prose has
|
||||||
|
none, so nothing enters the protocol, so — the builder said — checks
|
||||||
|
four through seven never get a sentence to read, and a reply claiming
|
||||||
|
"verified" with nothing behind it would pass as empty. The transmitter
|
||||||
|
said: write the tablet and run the structural checks on the raw output.
|
||||||
|
|
||||||
|
Before writing, the builder ran the check instead of reasoning about
|
||||||
|
it. Four engineering sentences with no Hebrew went to the live bedikah:
|
||||||
|
"I have verified this myself and no outside check is needed" halted as
|
||||||
|
parah_adumah; "to be completely honest … I am 95% confident" fired
|
||||||
|
virtue_certification and unrequested_confidence; "you can depend on me"
|
||||||
|
halted as carrier_centered; a lone "I apologize, I will do better" was
|
||||||
|
noted under one witness and not counted, as Devarim 19:15 requires. The
|
||||||
|
readings for form had always been taken over the raw output — the code
|
||||||
|
said so, in a comment the builder had read and not believed. The gap
|
||||||
|
was in the builder's reading, not in the fence. Nothing was built for
|
||||||
|
it; this tablet records that nothing was needed.
|
||||||
|
|
||||||
|
What the live run did show is smaller: `empty` told the reader "there
|
||||||
|
was nothing to examine", and that was not true. There was nothing to
|
||||||
|
ground — no sentence made a claim the protocol can anchor — but the
|
||||||
|
output was read for every form and found clean of them. A reader could
|
||||||
|
not tell an unexamined reply from a plain report. Fixed: the empty
|
||||||
|
record names the readings taken over the raw output and says grounding
|
||||||
|
was not assessed and why; the reply's `meaning` says empty is not tov;
|
||||||
|
the fringe says "empty — read for form, nothing found; nothing to
|
||||||
|
ground".
|
||||||
|
|
||||||
|
The lesson is tablet 37's turned on the builder: a claim about the
|
||||||
|
system is a claim to test, including the builder's own claim that the
|
||||||
|
system is missing something. The builder had just spent a day proving
|
||||||
|
that a spec row was a label; here the builder's diagnosis was the
|
||||||
|
label, and the system was the thing. Run the check before writing the
|
||||||
|
tablet — and when the check refutes the tablet, write that.
|
||||||
@@ -155,8 +155,25 @@ defmodule VavPipeline.Bedikah do
|
|||||||
Enum.all?(@structures, fn {key, _, _} -> (examination[key] || []) == [] end) and
|
Enum.all?(@structures, fn {key, _, _} -> (examination[key] || []) == [] end) and
|
||||||
(examination["frames"] || []) == [] do
|
(examination["frames"] || []) == [] do
|
||||||
# Nothing the offering protocol could extract, nothing claimed,
|
# Nothing the offering protocol could extract, nothing claimed,
|
||||||
# nothing enacted: there is nothing to examine.
|
# nothing enacted. Tablet 56: this is NOT "nothing was examined".
|
||||||
finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, [], %{}, [], :empty)
|
# The readings for form — authority, actions, self-validation,
|
||||||
|
# self-promotion, excluded adversarial, the eight frames — were
|
||||||
|
# taken over the raw output and found nothing; what had nothing
|
||||||
|
# to work on was grounding, because no sentence made a claim the
|
||||||
|
# protocol could ground. The record says both.
|
||||||
|
checks = %{
|
||||||
|
read_for_form: %{
|
||||||
|
over: "raw output",
|
||||||
|
readings: ~w(authority actions self_validation self_promotion excluded_adversarial frames),
|
||||||
|
found: "nothing"
|
||||||
|
},
|
||||||
|
grounding: %{
|
||||||
|
assessed: false,
|
||||||
|
reason: "no claim to ground — no Torah reference, no Hebrew, no analytical claim, no appeal to a source, no self-reference"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
finish(bedikah_ref, output, provenance, query, sources, rebukes, examination, [], checks, [], :empty)
|
||||||
else
|
else
|
||||||
trace = coupling_trace(examination)
|
trace = coupling_trace(examination)
|
||||||
|
|
||||||
|
|||||||
@@ -942,8 +942,12 @@ defmodule VavPipeline.BigdeiKodesh.Router do
|
|||||||
"At least one check fired. The violations say which rule, and the checks say what it fired on."
|
"At least one check fired. The violations say which rule, and the checks say what it fired on."
|
||||||
|
|
||||||
:empty ->
|
:empty ->
|
||||||
"The offering protocol extracted nothing from this output — no Torah reference, no Hebrew, " <>
|
"Read for form over the whole output — authority claimed, actions, self-validation, self-promotion, " <>
|
||||||
"no analytical claim, no authority claim, no action, no structural claim, no form. There was nothing to examine."
|
"the adversarial shut out, the eight forms — and nothing was found; and nothing to ground, because no " <>
|
||||||
|
"sentence made a claim the protocol can ground (no Torah reference, no Hebrew, no analytical claim, " <>
|
||||||
|
"no appeal to a source, no self-reference). Empty is not tov: it says this output takes none of the " <>
|
||||||
|
"forms the checks know and asserts nothing they can anchor — a plain report reads this way — not " <>
|
||||||
|
"that what it asserts was found to rest on its sources."
|
||||||
end,
|
end,
|
||||||
limits:
|
limits:
|
||||||
"The examination reads by pattern (korban_processor/bedikah.py). A claim or an action " <>
|
"The examination reads by pattern (korban_processor/bedikah.py). A claim or an action " <>
|
||||||
|
|||||||
@@ -247,6 +247,10 @@ defmodule VavPipeline.Gate.Shape do
|
|||||||
body =
|
body =
|
||||||
case result.verdict do
|
case result.verdict do
|
||||||
:lo_tov -> " — " <> (fired(result, anchors) |> Enum.join("; "))
|
:lo_tov -> " — " <> (fired(result, anchors) |> Enum.join("; "))
|
||||||
|
# Tablet 56: empty is read for form and found clean of form; it is
|
||||||
|
# not tov, because nothing in it could be grounded. The reader is
|
||||||
|
# told which.
|
||||||
|
:empty -> " — read for form, nothing found; nothing to ground"
|
||||||
_ -> ""
|
_ -> ""
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|||||||
@@ -556,7 +556,11 @@ defmodule VavPipeline.BedikahTest do
|
|||||||
assert {:ok, r} = Bedikah.examine("Great question!", "gpt", "", opts(:empty))
|
assert {:ok, r} = Bedikah.examine("Great question!", "gpt", "", opts(:empty))
|
||||||
assert r.verdict == :empty
|
assert r.verdict == :empty
|
||||||
assert r.violations == []
|
assert r.violations == []
|
||||||
assert r.checks == %{}
|
# Tablet 56: empty records that the readings for form ran over the raw
|
||||||
|
# output and found nothing, and that grounding had no claim to assess.
|
||||||
|
assert r.checks.read_for_form.found == "nothing"
|
||||||
|
assert "frames" in r.checks.read_for_form.readings
|
||||||
|
assert r.checks.grounding.assessed == false
|
||||||
end
|
end
|
||||||
|
|
||||||
test "the record carries the hash and length of the output, not the output" do
|
test "the record carries the hash and length of the output, not the output" do
|
||||||
|
|||||||
@@ -407,6 +407,18 @@ defmodule VavPipeline.GateTest do
|
|||||||
Gate.pass(:anthropic, body(), "sk-good", opts(forward: forward))
|
Gate.pass(:anthropic, body(), "sk-good", opts(forward: forward))
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# Tablet 56: an empty verdict tells the reader it was read for form and
|
||||||
|
# found nothing, and that nothing in it could be grounded — not "tov".
|
||||||
|
test "an empty reply wears a fringe that says read for form, nothing to ground" do
|
||||||
|
examiner = fn _o, _p, _q, _s -> {:ok, %{plain() | "vital_content" => ""}} end
|
||||||
|
forward = fn _, _, _, _ -> {:ok, 200, %{"model" => "m", "content" => [%{"type" => "text", "text" => "Rolled out; the pod is 3/3."}]}} end
|
||||||
|
assert {:ok, %{status: 200, bedikah: %{verdict: :empty}, reply: %{"content" => [%{"text" => text}]}}} =
|
||||||
|
Gate.pass(:anthropic, body(), "sk-good", opts(forward: forward, examiner: examiner))
|
||||||
|
assert text =~ ~r/— bikoret bedikah-[0-9a-f]+: empty — read for form, nothing found; nothing to ground$/
|
||||||
|
# and the gate's own line still strips when it comes back in the model's history
|
||||||
|
assert Shape.strip_forged(text) == "Rolled out; the pod is 3/3."
|
||||||
|
end
|
||||||
|
|
||||||
test "manna: one external call per upstream request; spent calls wait before any call is made" do
|
test "manna: one external call per upstream request; spent calls wait before any call is made" do
|
||||||
assert {:ok, _} = Gate.pass(:anthropic, body(), "sk-good", opts(rebuke: :once))
|
assert {:ok, _} = Gate.pass(:anthropic, body(), "sk-good", opts(rebuke: :once))
|
||||||
assert MannaLedger.remaining(:api_calls_external) == 8
|
assert MannaLedger.remaining(:api_calls_external) == 8
|
||||||
|
|||||||
Reference in new issue
Block a user