One gate to the world: Shorashim and Gitea behind the shared ingress

Three NLBs became one (185.19.28.170, ingress-nginx). Shorashim: `shorashim-external` (NLB 185.19.29.180, NodePort 30923) is replaced by an Ingress — a host-less catch-all (works by IP, no DNS) plus a `shorashim.ch` rule ready for the A record; TLS deliberately not requested until the name resolves. Sanhedrin-length proxy timeouts. The watch behind the ingress (completes tablet 46): the TCP peer is now the ingress controller, so Mishmar.peer/1 takes the caller from X-Real-IP — only when the peer is inside TRUSTED_PROXIES (the pod network, configmap); from anyone else the header is the caller's own claim and is ignored. The ingress Service was already externalTrafficPolicy: Local with its health-check port open, so the caller's address arrives without the security-group change the direct route needed. Verified live: the 11th knock is 429, a forged X-Real-IP is still 429, and in the same minute the old route (pod sees the node) was a different subject. Gitea: no name of its resolves, so it rides the controller's TCP passthrough on the same NLB — :3000 → gitea:80, :22 → gitea:22 (ingress-tcp-services.yaml), on the two NodePorts the security group already allowed. gitea-mcp now reaches Gitea over the cluster network instead of out through an NLB and back. Tamid §1.1 note, KOHEN.md, AGENTS.md. Doorpost e4e3d56d… → a2ec3842…. Elixir 539/0 (L3 on and off). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 17:59:23 +00:00
1 parent 54ca651f37
commit e17436c4ca
@@ -103,7 +103,8 @@ approve decisions, or supervise work. The specification does all of this.

 | Service | Location | Access |
 |---------|----------|--------|
-| Gitea | `http://185.19.29.94` | User: heaviness510 |
+| Gitea | `http://185.19.28.170:3000` (SSH :22) — behind the shared ingress NLB | User: heaviness510 |
+| Shorashim | `http://185.19.28.170` — behind the same ingress (catch-all; `shorashim.ch` once DNS points there) | keys: KOHEN.md "Issuing a key" |
 | PostgreSQL | Exoscale DBaaS ch-gva-2 | `temple-council-db` |
 | Kubernetes | Exoscale SKS ch-gva-2 | `~/.kube/exoscale-temple-council.yaml` |

@@ -73,7 +73,7 @@ You inherit infrastructure that has been provisioned and verified:

 | Item | Where |
 |------|-------|
-| Repo | `git clone http://185.19.29.94/heaviness510/shorashim.git` |
+| Repo | `git clone http://185.19.28.170:3000/heaviness510/shorashim.git` |
 | Kubeconfig | `~/.kube/exoscale-temple-council.yaml` |
 | `kubectl` | already installed |
 | `exo` CLI | already installed |
@@ -463,7 +463,7 @@ kubectl logs -n shorashim deploy/shorashim -c vav-pipeline -f

 If the seven days passed without halt events, without Shabbat violations, without manna exhaustion, without any pod restarts:

-1. **Point DNS** — `shorashim.ch` A record to the LoadBalancer external IP
+1. **Point DNS** — `shorashim.ch` A record to the ingress IP (`kubectl get svc -n ingress-nginx ingress-nginx-controller`; `185.19.28.170`). Then add the TLS block noted in `deploy/k8s/service.yaml` — until then `<host>` is that IP over plain HTTP
 2. **Open the pe'ah endpoint to the world** — the stranger approaches first
 3. **Observe the first real interaction** — a question from outside
 4. If *ki tov*: open courtyard and chatzer endpoints. The system is consecrated.
@@ -583,7 +583,7 @@ Three things are kept outside the mishkan repository, each for a Torah-grounded

 | Repo / directory | Where | Why |
 |---|---|---|
-| `shorashim-readings` | `http://185.19.29.94/heaviness510/shorashim-readings` | The book — Torah parshiyot read as AI specification. Companion to the mishkan. Lives beside the aron, not inside (Devarim 31:26). |
+| `shorashim-readings` | `http://185.19.28.170:3000/heaviness510/shorashim-readings` | The book — Torah parshiyot read as AI specification. Companion to the mishkan. Lives beside the aron, not inside (Devarim 31:26). |
 | `~/shorashim-source-data/morphhb` | local only | The quarry — raw WLC source. The carved tablets go in the aron; the quarry does not. |
 | `~/shorashim-operator-notes` | local only / private | The kohen's runbook — environment-specific details, hostnames, account names. Private operator notes. Not part of the system's specification. |

@@ -25,6 +25,12 @@ data:
  # keys never are.
  API_KEYS_FILE: "/etc/shorashim/secrets/api_keys"

+  # The pod network. The ingress controller lives in it; only from here
+  # is X-Real-IP believed as the caller's address (the pe'ah's per-IP
+  # watch, Tamid §1.1). Nothing outside the cluster can reach the pod
+  # directly any more (service.yaml), so nothing outside can forge it.
+  TRUSTED_PROXIES: "192.168.0.0/16"
+
  # Shoresh Engine binary — staged onto the shared shoresh-bin volume
  # by the shoresh-engine-install init container (see deployment.yaml).
  # vav_pipeline spawns it as an Erlang Port, i.e. as a child process of
@@ -3,8 +3,11 @@
 # Deploys the official Gitea MCP server (gitea.com/gitea/gitea-mcp)
 # in HTTP mode, exposing /mcp endpoint for Streamable HTTP transport.
 #
-# The MCP server connects to the Gitea backend at 185.19.29.94
-# using a Personal Access Token stored in gitea-mcp-secrets.
+# The MCP server connects to the Gitea backend over the cluster network
+# (gitea.temple-council.svc.cluster.local) using a Personal Access Token
+# stored in gitea-mcp-secrets. Until 2026-10-06 it went out to Gitea's
+# own NLB (185.19.29.94) and back in; that NLB is gone — Gitea is behind
+# the shared ingress now (ingress-tcp-services.yaml).
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -31,7 +34,7 @@ spec:
            - "-t"
            - "http"
            - "--host"
-            - "http://185.19.29.94"
+            - "http://gitea.temple-council.svc.cluster.local"
            - "--port"
            - "8080"
          env:
@@ -41,7 +44,7 @@ spec:
                  name: gitea-mcp-secrets
                  key: GITEA_TOKEN
            - name: GITEA_HOST
-              value: "http://185.19.29.94"
+              value: "http://gitea.temple-council.svc.cluster.local"
          ports:
            - containerPort: 8080
              name: http
@@ -3,11 +3,11 @@
 #
 # Two secrets needed:
 # 1. GITEA_TOKEN: Personal Access Token from Gitea
-#    Create at: http://185.19.29.94/user/settings/applications
+#    Create at: http://185.19.28.170:3000/user/settings/applications
 #    Permissions: repo (read/write)
 #
 # 2. OAuth Client ID/Secret: Registered OAuth app in Gitea
-#    Create at: http://185.19.29.94/user/settings/applications
+#    Create at: http://185.19.28.170:3000/user/settings/applications
 #    Redirect URI: https://claude.ai/api/mcp/auth_callback
 #    (These go into claude.ai connector settings, not this secret)
 ---
@@ -0,0 +1,38 @@
+# Gitea behind the shared ingress — TCP passthrough
+#
+# One NLB for the cluster (185.19.28.170, the ingress-nginx Service)
+# instead of one per Service. Shorashim is an HTTP Ingress
+# (service.yaml). Gitea has no hostname that resolves
+# (gitea.temple-council.ch is configured in Gitea and exists nowhere),
+# and host-based routing needs a name — so Gitea rides the ingress
+# controller's TCP passthrough on two ports of the same NLB:
+#
+#   185.19.28.170:3000  →  temple-council/gitea:80   (HTTP — git, web UI)
+#   185.19.28.170:22    →  temple-council/gitea:22   (SSH)
+#
+# Three live objects carry this; only the first is a whole manifest:
+#
+# 1. This ConfigMap.
+# 2. The controller reads it — one argument on the Deployment:
+#      kubectl -n ingress-nginx patch deploy ingress-nginx-controller --type=json -p \
+#        '[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
+#           "value":"--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services"}]'
+# 3. The controller's Service exposes the two ports ON THE NODEPORTS THE
+#    SECURITY GROUP ALREADY ALLOWS (they were Gitea's own; the CCM never
+#    opens a NodePort — tablets 29, 46):
+#      kubectl -n ingress-nginx patch svc ingress-nginx-controller --type=json -p \
+#        '[{"op":"add","path":"/spec/ports/-","value":{"name":"gitea-http","port":3000,"targetPort":3000,"protocol":"TCP","nodePort":30575}},
+#          {"op":"add","path":"/spec/ports/-","value":{"name":"gitea-ssh","port":22,"targetPort":22,"protocol":"TCP","nodePort":31345}}]'
+#    and temple-council/gitea is type ClusterIP (it held 30575/31345 as a
+#    LoadBalancer until 2026-10-06).
+#
+# When Gitea has a real name: add an HTTP Ingress for it (TLS via
+# cert-manager) and drop the 3000 line here; SSH stays.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: tcp-services
+  namespace: ingress-nginx
+data:
+  "3000": "temple-council/gitea:80"
+  "22": "temple-council/gitea:22"
@@ -24,38 +24,64 @@ spec:
      protocol: TCP
      name: http
 ---
-# External access via LoadBalancer
-# For Exoscale SKS, this provisions an Exoscale NLB.
+# ============================================================
+# The gate to the world — through the shared ingress
+# ============================================================
 #
-# The NLB forwards to this Service's NodePort and TCP-healthchecks it on
-# the node. The cloud controller does NOT open that port in the node's
-# security group — the operator must, or the NLB IP refuses connections
-# while the pod is Ready (2026-10-04, tablet 29):
-#   PORT=$(kubectl -n shorashim get svc shorashim-external -o jsonpath='{.spec.ports[0].nodePort}')
-#   exo compute security-group rule add default --flow ingress --protocol tcp --port $PORT --network 0.0.0.0/0
-apiVersion: v1
-kind: Service
+# Until 2026-10-06 Shorashim had its own LoadBalancer Service
+# (`shorashim-external`, NLB 185.19.29.180, NodePort 30923). It is gone:
+# one NLB per Service was a third of the account's bill, and the pod saw
+# every caller as the node (kube-proxy SNAT) so the pe'ah's per-IP watch
+# could not tell callers apart. The ingress-nginx Service is already
+# externalTrafficPolicy: Local with its health-check port open, so the
+# controller sees the caller and hands it on in X-Real-IP; the pipeline
+# believes that header only from TRUSTED_PROXIES (configmap.yaml).
+#
+# Two rules, one backend:
+#   - no host  → the catch-all: http://<ingress-ip>/... works with no DNS
+#   - shorashim.ch → ready for the A record (KOHEN.md Day 8). TLS is NOT
+#     requested here: asking cert-manager for a certificate before the
+#     name resolves only burns Let's Encrypt's failed-validation limit.
+#     When the record exists add:
+#       metadata.annotations:
+#         cert-manager.io/cluster-issuer: letsencrypt-prod
+#         nginx.ingress.kubernetes.io/ssl-redirect: "true"
+#       spec.tls: [{hosts: [shorashim.ch], secretName: shorashim-tls}]
+#     and point callers at https://shorashim.ch — until then the
+#     courtyard keys travel over plain HTTP.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
 metadata:
-  name: shorashim-external
+  name: shorashim
  namespace: shorashim
  labels:
    app.kubernetes.io/name: shorashim
  annotations:
-    # Exoscale NLB annotations
-    service.beta.kubernetes.io/exoscale-loadbalancer-zone: "ch-gva-2"
-    service.beta.kubernetes.io/exoscale-loadbalancer-name: "shorashim"
+    # A 71-judge Sanhedrin may deliberate for ten minutes (Construction §2.2).
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "620"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "620"
+    # The bedikah accepts 65,536 bytes of output; leave room for the envelope.
+    nginx.ingress.kubernetes.io/proxy-body-size: "256k"
+    nginx.ingress.kubernetes.io/ssl-redirect: "false"
 spec:
-  type: LoadBalancer
-  # Local: kube-proxy does not SNAT, so the pod sees the caller's own
-  # address and the pe'ah's per-IP watch (Tamid §1.1) counts callers,
-  # not the node. With Cluster (the default) every caller arrived as the
-  # node IP 91.92.153.113 — observed 2026-10-06, tablet 46. One node,
-  # one replica: Local loses nothing.
-  externalTrafficPolicy: Local
-  selector:
-    app.kubernetes.io/name: shorashim
-  ports:
-    - port: 80
-      targetPort: 4000
-      protocol: TCP
-      name: http
+  ingressClassName: nginx
+  rules:
+    - http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: shorashim
+                port:
+                  number: 80
+    - host: shorashim.ch
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: shorashim
+                port:
+                  number: 80
@@ -48,8 +48,10 @@ Rate limit: 10 requests per minute per IP
           No identification stored. IP is used for rate-limiting only,
           not logged beyond the current request.
           429 carries Retry-After and the manna explanation.
-           (The Service must deliver the caller's address to the pod —
-           externalTrafficPolicy: Local — or every caller is one IP.)
+           (The caller's address reaches the pod through the ingress:
+           the controller's Service is externalTrafficPolicy: Local and
+           it writes X-Real-IP, believed only from the pod network.
+           Without both, every caller is one IP.)
 ```

 **Design requirements:** No cookies. No session state. No analytics. No tracking pixels. No JavaScript fingerprinting. The pe'ah endpoint collects nothing. The stranger is safe to approach.
@@ -27,7 +27,10 @@ config :vav_pipeline,
    peah: [minute: 10, hour: 100],
    courtyard: [minute: 20, hour: 200],
    chatzer: [minute: 5, hour: 60]
-  ]
+  ],
+  # Peers whose X-Real-IP is believed (the ingress controller). CIDRs.
+  # Empty → the TCP peer is the caller.
+  trusted_proxies: []

 # L3 — Oral Tradition (PostgreSQL)
 # Architecture §3.7: Append-only. INSERT allowed. Never UPDATE. Never DELETE.
@@ -32,6 +32,12 @@ if config_env() == :prod do
      System.get_env("KORBAN_SOCKET_PATH") || "/tmp/shorashim/korban.sock",
    # The keys themselves are in the file (a mounted Secret), not in env.
    api_keys_file: System.get_env("API_KEYS_FILE") || "/etc/shorashim/secrets/api_keys",
+    # The ingress controller's network: only from there is X-Real-IP the
+    # caller's address (BigdeiKodesh.Mishmar.peer/1). Comma-separated CIDRs.
+    trusted_proxies:
+      (System.get_env("TRUSTED_PROXIES") || "")
+      |> String.split(",", trim: true)
+      |> Enum.map(&String.trim/1),
    # Shoresh Engine binary — VavPipeline.ShoreshEnginePort opens an
    # Erlang Port with {:spawn_executable, binary}, which spawns a CHILD
    # PROCESS OF THIS CONTAINER. A Port cannot cross a container
@@ -107,15 +107,68 @@ defmodule VavPipeline.BigdeiKodesh.Mishmar do
    end
  end

-  @doc "The subject for a request at the pe'ah: its peer address, as text."
+  @doc """
+  The subject for a request at the pe'ah: the caller's address, as text.
+
+  Behind the ingress the TCP peer is the ingress controller, not the
+  caller; the controller writes the caller's address in `X-Real-IP`
+  (it overwrites whatever the caller sent). That header is believed
+  ONLY when the peer is inside `:trusted_proxies` (CIDRs — the pod
+  network). From any other peer the header is the caller's own claim
+  and is ignored: otherwise a caller could name a fresh address with
+  every knock and never be counted.
+  """
  @spec peer(Plug.Conn.t()) :: subject()
-  def peer(%Plug.Conn{remote_ip: ip}) do
+  def peer(%Plug.Conn{remote_ip: ip} = conn) do
+    with true <- trusted_proxy?(ip),
+         [claimed] <- Plug.Conn.get_req_header(conn, "x-real-ip"),
+         {:ok, real} <- claimed |> String.trim() |> String.to_charlist() |> :inet.parse_address() do
+      ntoa(real)
+    else
+      _ -> ntoa(ip)
+    end
+  end
+
+  @doc "Is this peer one of the configured proxies (`:trusted_proxies`, a list of CIDR strings)?"
+  @spec trusted_proxy?(:inet.ip_address()) :: boolean()
+  def trusted_proxy?(ip) do
+    :vav_pipeline
+    |> Application.get_env(:trusted_proxies, [])
+    |> Enum.any?(&in_cidr?(ip, &1))
+  end
+
+  defp ntoa(ip) do
    case :inet.ntoa(ip) do
      {:error, _} -> inspect(ip)
      chars -> List.to_string(chars)
    end
  end

+  # "192.168.0.0/16" — a malformed entry matches nothing (and so trusts
+  # nobody), which is the safe way for a typo to fail.
+  defp in_cidr?(ip, cidr) when is_binary(cidr) do
+    with [net, bits] <- String.split(String.trim(cidr), "/", parts: 2),
+         {:ok, net_ip} <- net |> String.to_charlist() |> :inet.parse_address(),
+         {prefix, ""} <- Integer.parse(bits),
+         true <- tuple_size(net_ip) == tuple_size(ip) do
+      width = if tuple_size(ip) == 4, do: 8, else: 16
+      total = width * tuple_size(ip)
+
+      if prefix in 0..total do
+        shift = total - prefix
+        Bitwise.bsr(to_int(ip, width), shift) == Bitwise.bsr(to_int(net_ip, width), shift)
+      else
+        false
+      end
+    else
+      _ -> false
+    end
+  end
+
+  defp to_int(ip, width) do
+    ip |> Tuple.to_list() |> Enum.reduce(0, fn part, acc -> Bitwise.bsl(acc, width) + part end)
+  end
+
  @doc "Forget every window. For tests."
  def reset(server \\ __MODULE__), do: GenServer.call(server, :reset)

@@ -175,7 +175,7 @@ defmodule VavPipeline.Doorpost do
  # changes; the update is a deliberate operator commit. A bare
  # hash-update commit (without a corresponding spec change in the
  # same commit) is itself a drift signal.
-  @expected_hash "e4e3d56de7422280b5562c4a7ebce02fe05225a938bbce26d6e52f26b6b240eb"
+  @expected_hash "a2ec3842294a7ffd84337aa21e9c5293ba39f70c17134d4edaaa97f207b8ae5e"

  @doc "Return the list of files that constitute the category-grade specification."
  @spec files() :: [String.t()]
@@ -88,6 +88,71 @@ defmodule VavPipeline.Tablet46MishmarTest do
    end
  end

+  describe "Mishmar.peer/1 — who is knocking, behind the ingress" do
+    setup do
+      original = Application.get_env(:vav_pipeline, :trusted_proxies)
+      Application.put_env(:vav_pipeline, :trusted_proxies, ["192.168.0.0/16", "not-a-cidr", "fd00::/8"])
+      on_exit(fn -> Application.put_env(:vav_pipeline, :trusted_proxies, original) end)
+      :ok
+    end
+
+    test "from the ingress, X-Real-IP is the caller" do
+      conn = %{conn(:post, "/peah/ask") | remote_ip: {192, 168, 150, 211}} |> put_req_header("x-real-ip", "203.0.113.7")
+      assert Mishmar.peer(conn) == "203.0.113.7"
+
+      v6 = %{conn(:post, "/peah/ask") | remote_ip: {0xFD00, 0, 0, 0, 0, 0, 0, 1}} |> put_req_header("x-real-ip", "2001:db8::7")
+      assert Mishmar.peer(v6) == "2001:db8::7"
+    end
+
+    test "from anyone else, X-Real-IP is only a claim — the peer is the caller" do
+      conn = %{conn(:post, "/peah/ask") | remote_ip: {203, 0, 113, 9}} |> put_req_header("x-real-ip", "198.51.100.1")
+      assert Mishmar.peer(conn) == "203.0.113.9"
+
+      # Just outside the /16.
+      edge = %{conn(:post, "/peah/ask") | remote_ip: {192, 169, 0, 1}} |> put_req_header("x-real-ip", "198.51.100.1")
+      assert Mishmar.peer(edge) == "192.169.0.1"
+    end
+
+    test "from the ingress without the header, or with a header that is not an address — the peer" do
+      bare = %{conn(:post, "/peah/ask") | remote_ip: {192, 168, 1, 1}}
+      assert Mishmar.peer(bare) == "192.168.1.1"
+
+      junk = bare |> put_req_header("x-real-ip", "'; drop table")
+      assert Mishmar.peer(junk) == "192.168.1.1"
+    end
+
+    test "no trusted proxies configured — nobody's header is believed" do
+      Application.put_env(:vav_pipeline, :trusted_proxies, [])
+      conn = %{conn(:post, "/peah/ask") | remote_ip: {192, 168, 150, 211}} |> put_req_header("x-real-ip", "203.0.113.7")
+      assert Mishmar.peer(conn) == "192.168.150.211"
+    end
+
+    test "a caller cannot escape the watch by naming a new address with every knock" do
+      knock = fn claimed ->
+        %{conn(:post, "/peah/ask", Jason.encode!(%{question: "x"})) | remote_ip: {203, 0, 113, 50}}
+        |> put_req_header("content-type", "application/json")
+        |> put_req_header("x-real-ip", claimed)
+        |> Router.call(@opts)
+      end
+
+      statuses = for i <- 1..4, do: knock.("198.51.100.#{i}").status
+      assert List.last(statuses) == 429
+    end
+
+    test "two callers through the same ingress are counted apart" do
+      via = fn real ->
+        %{conn(:post, "/peah/ask", Jason.encode!(%{question: "x"})) | remote_ip: {192, 168, 150, 211}}
+        |> put_req_header("content-type", "application/json")
+        |> put_req_header("x-real-ip", real)
+        |> Router.call(@opts)
+      end
+
+      for _ <- 1..3, do: via.("203.0.113.60")
+      assert via.("203.0.113.60").status == 429
+      refute via.("203.0.113.61").status == 429
+    end
+  end
+
  describe "at the router" do
    test "the pe'ah: the fourth knock from one address in a minute is 429 with Retry-After and the manna explanation" do
      statuses = for _ <- 1..4, do: ask({203, 0, 113, 9}).status