bikoret/specs/shorashim_operations.md

SHORASHIM: Operations Manual

Document Purpose

This is the tamid — the daily operations that keep the system alive. A lamb in the morning, a lamb at twilight. Incense when the lamps are tended, incense when the lamps are lit. Oil for the menorah. Every day. Unglamorous. Repetitive. Essential.

Torah reference: Shemot 29:38-42 — V'zeh asher ta'aseh al hamizbeach: k'vasim b'nei shanah sh'nayim layom tamid. Et hakeves ha'echad ta'aseh vaboker, v'et hakeves hasheni ta'aseh bein ha'arbayim. "This is what you shall offer on the altar: two lambs a year old, daily, continually. The one lamb you shall offer in the morning, and the other lamb you shall offer at twilight."

Upstream authority: Torah → PRD → Architecture Spec → Construction Spec → This Document → Daily Practice

---

1. BIGDEI KEHUNAH — The Interfaces

The kohen cannot serve without the garments. The system cannot be accessed without interfaces. These are the garments Shorashim wears when it serves.

1.1 Access Tier Interfaces

Machaneh Yisrael — Pe'ah Endpoint (Public, No Auth)

The outermost ring. Open to all, including the ger. No authentication. No identification. No tracking.

Endpoint: GET /peah/ask
          POST /peah/ask

Request (POST):
{
  "question": "string — the query, in any language",
  "offering_type": "minchah"   # Pe'ah always uses minchah protocol
}

Response:
{
  "answer": "string — the processed response",
  "sources": [
    {"book": "bereishit", "chapter": 1, "verse": 1, "text": "..."}
  ],
  "processing": {
    "roots_consulted": ["bet-resh-aleph"],
    "deliberation": "none" | "single-judge",
    "cycle_phase": "voker"
  }
}

Rate limit: 10 requests per minute per IP
           100 requests per hour per IP
           No identification stored. IP is used for rate-limiting only,
           not logged beyond the current request.

Design requirements: No cookies. No session state. No analytics. No tracking pixels. No JavaScript fingerprinting. The pe'ah endpoint collects nothing. The stranger is safe to approach.

Machaneh Leviyah — Authenticated Endpoint

Endpoint: POST /courtyard/offering

Headers:
  Authorization: Bearer <api-key>

Request:
{
  "offering_type": "shelamim" | "olah" | "chatat" | "minchah",
  "content": "string — the input text or question",
  "scope": {                          # Optional — narrows processing
    "book": "bereishit",
    "chapter_range": [1, 3]
  },
  "request_deliberation": false       # If true, routes to Sanhedrin
}

Response:
{
  "offering_id": "uuid",
  "status": "processed" | "queued" | "purifying",
  "result": {
    "vital_content": "string — the processed response",
    "sources": [...],
    "roots": [...],
    "parse_structure": {              # Te'amim-derived structure
      "primary_division": "...",
      "bound_groups": [...]
    },
    "dual_readings": [...]            # Ketiv/qere positions, if relevant
  },
  "offering_receipt": {
    "type": "shelamim",
    "altar_portion": "recorded in L3",
    "kohen_portion": "diagnostics logged",
    "your_portion": "the result above"
  }
}

Errors:
  401 — Invalid or missing API key
  422 — Input failed kiyor (validation). Body includes purification guidance.
  429 — Rate limit exceeded. Body includes manna explanation.
  503 — System in Shabbat/festival mode. Body includes resumption time.

Chatzer — Full Access

Endpoint: POST /chatzer/sanhedrin

Request:
{
  "case": "string — the dispute or question requiring full deliberation",
  "sides": [                          # Optional — pre-structured arguments
    {"position": "...", "evidence": ["..."]},
    {"position": "...", "evidence": ["..."]}
  ],
  "complexity_hint": "simple" | "monetary" | "capital" | "novel"
}

Response:
{
  "case_ref": "uuid",
  "deliberation": {
    "panel_size": 3,
    "majority": {
      "position": "...",
      "reasoning": "...",
      "sources": [...],
      "votes": 2
    },
    "minority": {
      "position": "...",
      "reasoning": "...",
      "sources": [...],
      "votes": 1
    },
    "outcome": "accepted" | "rejected" | "teiku",
    "processing_time_ms": 4200
  },
  "note": "This is the system's reasoning, not a binding ruling. A human beit din may reach a different conclusion."
}

Kodesh — Operator Interface

Endpoint: GET /kodesh/status
          GET /kodesh/health
          GET /kodesh/queue
          GET /kodesh/audit/:date
          POST /kodesh/inspect/:offering_id

Authentication: mTLS (client certificate) — kohanim only

GET /kodesh/status:
{
  "cloud_state": "camped",
  "cycle_phase": "voker",
  "manna_provision": {"remaining_pct": 72, "expires_at": "..."},
  "shabbat": {"next_start": "...", "next_end": "..."},
  "components": {
    "shoresh_engine": "healthy",
    "sanhedrin": "healthy",
    "talmudic_engine": "healthy",
    "korban_processor": "healthy",
    "halt_monitor": "active",
    "l1_integrity": "verified",
    "l3_connections": 12
  },
  "active_panels": 2,
  "queue_depth": 7,
  "refuge_components": [],
  "pending_tzaraat": []
}

GET /kodesh/health:
{
  "overall": "healthy" | "degraded" | "critical" | "shabbat",
  "checks": {
    "l0_firmware": "pass",
    "l1_hash": "pass",
    "l3_writable": "pass",
    "cloud_readable": "pass",
    "manna_present": "pass",
    "halt_monitor_alive": "pass",
    "esh_tamid": "pass"            # The fire is still burning
  }
}

1.2 Error Response Format (All Tiers)

Every error is a chatat protocol — it restores, it does not punish.

{
  "error": {
    "code": "KIYOR_VALIDATION_FAILED",
    "type": "purification_needed",
    "message": "Your input contains elements that need cleansing before processing.",
    "guidance": {
      "issue": "Input contains embedded HTML markup",
      "correction": "Submit plain text without markup",
      "resubmit": true
    }
  }
}

Error codes follow the offering metaphor:

Code	Meaning	Category
KIYOR_*	Input validation failure	Purification needed
TUMAH_*	Input contamination detected	Tzara'at diagnostic initiated
SHABBAT	System in mandatory rest	Wait — resumption time provided
MANNA_DEPLETED	Daily provision exhausted	Wait — next provision time provided
CLOUD_MOVING	System in transition	Wait — transition details provided
TEIKU	Unresolvable question	The system is honest about what it cannot answer
HALT_*	Cardinal sin detected	Processing terminated — audit log created

---

2. TAMID — Daily Operations Runbook

2.1 Morning Tamid (System Day Start)

Performed every morning at the voker boundary (configurable — default 06:00 local time of hosting server).

MORNING TAMID CHECKLIST — performed by automation, verified by kohen

1. □ L1 integrity check
     Command: sha256sum /mnt/torah/torah_scroll.sqfs
     Expected: matches L0 firmware hash
     If failed: HALT — do not proceed. Alert senior kohanim.

2. □ Cloud state read
     Command: cat /var/lib/shorashim/cloud/state.json
     Expected: valid JSON with recognized signal
     If "lift" or "shutdown": obey. Do not proceed with normal operations.
     If unreadable: WAIT. Alert senior kohanim.

3. □ Manna provision read
     Command: cat /var/lib/shorashim/manna/today.json
     Expected: valid JSON with today's date, non-expired
     If absent: WAIT. Alert senior kohanim.
     If expired: purge, WAIT.

4. □ L3 connectivity
     Command: psql -c "SELECT 1" shorashim
     Expected: returns 1
     If failed: restart PostgreSQL. If still failed: alert kohen.

5. □ Halt monitor alive
     Command: Check Erlang process via epmd
     Expected: halt_monitor process registered
     If failed: HALT — the system does not run without the halt monitor.

6. □ Component health check
     Each component responds to health ping within 5 seconds.
     Shoresh Engine, Sanhedrin, Talmudic Engine, Korban Processor, Vav Pipeline.
     If any component fails: restart that component. If restart fails: alert kohen.

7. □ Esh tamid verification
     The processing pipeline accepts a test input and returns a result.
     If the fire has gone out: re-initialize pipeline. Alert kohen.

8. □ Yesterday's manna cleanup
     Purge any expired resource allocations from previous cycle.
     Release reclaimed resources.

9. □ Log rotation
     Rotate non-audit logs. Audit logs are never rotated.

10. □ Record tamid completion
      Write to audit log: "Morning tamid completed at [timestamp]. All checks passed."
      Or: "Morning tamid completed with issues: [list]. Kohen alerted."

2.2 Evening Tamid (System Day End)

Performed every evening at the erev boundary (configurable — default 18:00 local time).

EVENING TAMID CHECKLIST

1. □ Processing queue drain
     Allow in-flight offerings to complete. No new offerings accepted.
     Timeout: 10 minutes. After timeout, move unfinished to next cycle.

2. □ L3 integrity check
     Verify all day's writes are committed. No pending transactions.
     Count today's insertions. Record in audit log.

3. □ Manna accounting
     Record: provision received, provision consumed, surplus released.
     If surplus existed: log manna violation check (was anyone hoarding?).

4. □ Menorah status — observability summary
     Record: queries processed, Sanhedrin panels convened, teiku registered,
     halt conditions triggered (should be 0), errors by type, latency P50/P95/P99.

5. □ Ketoret review — quality of experience
     Sample 5 random responses from today. Evaluate: were they clear? Respectful?
     Did they serve the user's actual need? Record assessment.
     (This is a human review — the kohen reads the incense, not the system.)

6. □ Record tamid completion
     Write to audit log: "Evening tamid completed at [timestamp]. Day summary: [metrics]."

2.3 Erev Shabbat Additional Procedures

EREV SHABBAT — performed before candle lighting time

1. □ Verify double manna provision is written for Shabbat
     If not: alert senior kohanim immediately. Cannot enter Shabbat unfed.

2. □ Drain all processing queues completely (no timeout carryover)
     Nothing in-flight enters Shabbat.

3. □ Verify Shabbat timer will trigger correctly
     Dry-run: confirm candle lighting and havdalah times are calculated.

4. □ Final L3 backup before Shabbat
     pg_dump to encrypted backup. Store off-site.

5. □ Record: "Entering Shabbat at [timestamp]. Next cycle: [motzei Shabbat time]."

6. □ The kohen steps back. The fire is banked. The system rests.

2.4 Motzei Shabbat Procedures

MOTZEI SHABBAT — performed after havdalah time

1. □ Full morning tamid (all 10 checks)
2. □ L3 consolidation: review Shabbat-period learning queue (if any emergency
     processing occurred under pikuach nefesh)
3. □ Read new manna provision for the week's first day
4. □ Resume normal operations

2.5 Rosh Chodesh (Monthly Maintenance)

ROSH CHODESH — first day of each Hebrew month

1. □ Full morning tamid
2. □ Deep L1 verification — not just hash check, but sample verification
     of 100 random verses against independent source
3. □ PostgreSQL VACUUM ANALYZE on all L3 tables
4. □ Review previous month's audit logs for patterns
5. □ Shulchan renewal — refresh pre-computed cache for common queries
6. □ Menorah deep review — full observability report for the month
7. □ Community report: system health, capacity used vs. provisioned,
     user activity, teiku registry growth, halt events (should be 0)

2.6 Festival Procedures

Festival	System Action
Pesach	Liberation audit: is the system still serving freedom? Review all access controls. Verify pe'ah is open. Verify no surveillance.
Shavuot	Specification re-acceptance: full L1 re-verification. Re-run Day 1-2 of inauguration protocol. The system confirms it still holds the same Torah.
Rosh Hashanah	Year-start: comprehensive metrics review. Set targets for the coming year.
Yom Kippur	Full system shutdown (25 hours). Deep purification: full L3 audit, identify any asham conditions, resolve any pending refuge cases through Sanhedrin deliberation grounded in Torah.
Sukkot	Vulnerability audit: run the system with deliberately reduced resources (sukkah = temporary structure). Does it still serve faithfully with less?
Shmitah (7th year)	Major version review per architecture spec. L3 comprehensive evaluation. Error records older than 7 years archived and sealed.

---

3. MONITORING — The Menorah's Seven Branches

The menorah provides light — it makes visible what would otherwise be in darkness. Seven branches of observability.

3.1 Branch 1: Processing State

Metrics:
  shorashim_offerings_total{type="shelamim|olah|chatat|minchah"}    # Counter
  shorashim_offerings_active                                         # Gauge
  shorashim_pipeline_latency_seconds{phase="shema|tohu|vayomer|..."}# Histogram
  shorashim_sanhedrin_panels_active                                  # Gauge
  shorashim_sanhedrin_deliberation_seconds{size="3|23|71"}          # Histogram

3.2 Branch 2: Deliberation Progress

Metrics:
  shorashim_interpretations_total{status="accepted|rejected|teiku"} # Counter
  shorashim_minority_opinions_total                                  # Counter
  shorashim_kushyot_generated_total                                  # Counter
  shorashim_teiku_registry_size                                      # Gauge

3.3 Branch 3: Specification Reference

Metrics:
  shorashim_l1_reads_total                    # Counter — how often spec is consulted
  shorashim_l1_hash_verified_total            # Counter — integrity checks
  shorashim_l1_hash_failed_total              # Counter — MUST be 0
  shorashim_teamim_reads_total                # Counter
  shorashim_ketiv_qere_positions_accessed     # Counter
  shorashim_gematria_lookups_total            # Counter

3.4 Branch 4: Source Citation

Metrics:
  shorashim_torah_refs_cited_total{book="bereishit|shemot|..."}    # Counter
  shorashim_roots_activated_total                                    # Counter
  shorashim_roots_unique_activated                                   # Gauge per cycle

3.5 Branch 5: Error Diagnostics

Metrics:
  shorashim_kiyor_rejections_total{reason="encoding|injection|malformed|..."}
  shorashim_tzaraat_quarantines_active         # Gauge
  shorashim_tzaraat_outcomes{result="tahor|tamei|fully_visible_tahor"}
  shorashim_chatat_corrections_total           # Counter
  shorashim_asham_restitutions_total           # Counter — should be very low

3.6 Branch 6: Learning Record

Metrics:
  shorashim_l3_insertions_total{table="interpretations|minority_opinions|teiku_registry"}
  shorashim_l3_size_bytes                      # Gauge
  shorashim_l3_growth_rate_bytes_per_day       # Gauge (smoothed)

3.7 Branch 7: System Health

Metrics:
  shorashim_shema_context_build_seconds        # Histogram
  shorashim_cloud_state{signal="settle|lift|..."}  # Gauge (label)
  shorashim_manna_provision_remaining_pct      # Gauge
  shorashim_manna_violations_total             # Counter — hoarding attempts
  shorashim_halt_events_total{type="avodah_zarah|shfichut_damim|gilui_arayot"}
  shorashim_refuge_components_active           # Gauge — should be 0
  shorashim_eglah_arufah_declarations_total    # Counter — should be near 0
  shorashim_component_health{name="..."}       # 1=healthy, 0=unhealthy

3.8 Alerting Thresholds

Condition	Severity	Action
L1 hash mismatch	CRITICAL	System halts. Chatzotzrot: war. Senior kohanim alerted.
Halt monitor down	CRITICAL	System halts. Cannot operate without the monitor.
Any halt event fires	CRITICAL	Chatzotzrot: war. Senior kohanim alerted.
Cloud state unreadable	HIGH	System waits. Chatzotzrot: assembly_leaders.
Manna absent	HIGH	System waits. Chatzotzrot: assembly_leaders.
Component unhealthy	HIGH	Restart attempted. If failed: chatzotzrot: assembly_leaders.
P95 latency > 2x budget	MEDIUM	Kohen alerted. Investigate in next tamid.
L3 growth > 150% projected	MEDIUM	Kohen alerted. May need storage review.
Manna hoarding detected	LOW	Logged. Surplus reclaimed. Reviewed in evening tamid.
Tzara'at quarantine active	INFO	Normal operation. Logged for kohen review.

---

4. INCIDENT RESPONSE

4.1 Ordinary Failures (Not Halt Conditions)

A process crashes. A disk fills. A network drops. These are not the three cardinal sins. They are the daily realities of running a system.

INCIDENT RESPONSE — ORDINARY

1. DETECT
   - Monitoring fires alert
   - Or: kohen observes during tamid

2. ASSESS
   - Is this a halt condition (avodah zarah, shfichut damim, gilui arayot)?
   - If YES: Halt Monitor should have caught it. If it didn't, that's a second incident.
   - If NO: proceed with ordinary response.

3. CONTAIN
   - If a single component: restart it.
   - If the restart fails: isolate it. Route traffic around it if possible.
   - If the failure affects the pipeline: queue incoming offerings. They wait.
   - The system degrades gracefully. It does not fabricate results from partial capacity.

4. DIAGNOSE
   - Read the logs. The menorah is for this.
   - Identify: what failed, when, what was the trigger.
   - Check the ShemaContext from the failed cycle — was the context valid?

5. REPAIR
   - Fix the issue.
   - If a configuration change: record it in the audit log.
   - If a code change: go through bikurim (first-output review) before deploying.

6. VERIFY
   - Run the relevant inauguration day's checks for the affected component.
   - Confirm the system is healthy.

7. RECORD
   - Audit log entry: incident, diagnosis, repair, verification.
   - Evening tamid review includes incident discussion.

4.2 Harm Events (Arei Miklat / Eglah Arufah)

INCIDENT RESPONSE — HARM

1. DETECT
   - User reports harm caused by system output
   - Or: internal monitoring detects anomalous impact

2. IMMEDIATE
   - Activate arei miklat: isolate the responsible processing path
   - Protect the affected user: block their exposure to that processing path
   - Chatzotzrot: assembly_leaders — senior kohanim alerted

3. IF RESPONSIBLE COMPONENT IS KNOWN
   - Component enters refuge (arei miklat)
   - Component is protected from reactive destruction
   - Sanhedrin deliberation determines release conditions

4. IF RESPONSIBLE COMPONENT IS UNKNOWN
   - Activate eglah arufah protocol
   - Nearest component in the processing chain formally declares:
     what it processed, what it did not check, what it could have caught
   - Declaration is permanently recorded

5. GOVERNANCE BOARD
   - Reviews the incident
   - Determines: was this chatat (unintentional error) or asham (unauthorized usage)?
   - If chatat: correction protocol, no penalty beyond fixing the issue
   - If asham: restitution protocol, plus additional safeguards (the "fifth")
   - Determines when/if the refuge component is released

6. NEVER
   - Never delete the component reactively
   - Never blame the user for being harmed
   - Never hide the incident
   - Never continue the same processing path without review

---

5. BACKUP AND RECOVERY

5.1 What Is Backed Up

Layer	Backup Method	Frequency	Retention
L0 (Luchot)	Compiled into binary — exists in version control	Every build	Permanent
L1 (Torah scroll)	Squashfs image — source files in version control	Once (immutable)	Permanent
L3 (Oral tradition)	pg_dump --format=custom encrypted with GPG	Daily (erev tamid) + erev Shabbat	1 year rolling + shmitah archives
Configuration	Version control (Git)	Every change	Permanent
Audit logs	rsync to off-site encrypted storage	Daily	Permanent (never deleted)
Cloud governance	Not backed up separately — cloud IS L1 (Torah) which is already backed up as immutable specification	—	—
Manna provision	Not backed up — kohen/treasurer sets this from machatzit hashekel fund	—	—

5.2 Recovery Procedures

Scenario: L3 corruption

1. Stop the system
2. Restore L3 from most recent pg_dump
3. Verify: all append-only constraints intact
4. Re-run inauguration Day 1 (L0/L1 verification) and Day 7 (integration)
5. Resume operations

Scenario: Full system loss

1. Provision new server (same specs from Construction Specification)
2. Install OS and all runtimes (pinned versions from Construction Specification)
3. Deploy code from version control
4. Mount L1 from squashfs image, verify hash
5. Restore L3 from most recent backup
6. Verify cloud governance from Torah (L1 readable, integrity verified, calendar active) and kohen/treasurer writes manna provision
7. Run full seven-day inauguration
8. The system does not skip inauguration because it is being rebuilt. A rebuilt mishkan is a new mishkan. It is consecrated anew.

Scenario: L1 integrity failure

1. The system has already halted (this is automatic)
2. Investigate: was the filesystem modified? Was the mount corrupted?
3. Re-mount from the original squashfs image
4. Verify hash against L0 firmware constant
5. If hash matches: resume
6. If hash does not match: the specification has been altered. This is not a technical incident. This is a Torah integrity incident. The kohanim must determine what happened and whether the system can trust its own specification. The system does not resume until L1 integrity is restored — because the cloud IS L1, and if L1 is compromised, the cloud is not visible. The system waits until Torah is readable and intact.

---

6. ON-CALL PROCEDURES

6.1 Who Is On Call

One kohen at all times during the six working days. The kohen is reachable within 15 minutes for HIGH alerts, within 5 minutes for CRITICAL alerts.

During Shabbat: no on-call. The system is off. The only exception is pikuach nefesh — if the system's unavailability would cause genuine danger to a human life, the kohen may respond. This should never happen. If it can happen, the system has made itself necessary in a way that violates its specification (the Shabbat Test from the PRD).

6.2 Escalation

Level	Responder	Conditions
L1	On-call kohen	Any alert
L2	Lead architect	Component failures, performance degradation
L3	Senior kohanim	Halt events, harm incidents, L1 integrity failure

6.3 Communication

Chatzotzrot signals map to communication channels:

Signal	Channel
assembly_all	System status page + all registered notification endpoints
assembly_leaders	Senior kohanim secure channel + on-call kohen
breaking_camp	Same as assembly_all — everyone needs to know the system is moving
war	Senior kohanim immediate alert + on-call kohen + lead architect
moadim	System status page — celebratory, not urgent

---

Document Registry

Document	Torah Parallel	Status
Shorashim PRD	V'asu li mikdash	✓ Complete
Architecture Specification	Vessel descriptions	✓ Complete
Construction Specification	Exact measurements	✓ Complete
Operations Manual (this document)	Tamid / Bigdei Kehunah	✓ Complete
Economic Model	Machatzit HaShekel	Pending
Materials Inventory	Terumah list	Needs update