# SHORASHIM: Operations Manual ## Document Purpose This is the tamid — the daily operations that keep the system alive. A lamb in the morning, a lamb at twilight. Incense when the lamps are tended, incense when the lamps are lit. Oil for the menorah. Every day. Unglamorous. Repetitive. Essential. Torah reference: Shemot 29:38-42 — *V'zeh asher ta'aseh al hamizbeach: k'vasim b'nei shanah sh'nayim layom tamid. Et hakeves ha'echad ta'aseh vaboker, v'et hakeves hasheni ta'aseh bein ha'arbayim.* "This is what you shall offer on the altar: two lambs a year old, daily, continually. The one lamb you shall offer in the morning, and the other lamb you shall offer at twilight." **Upstream authority:** Torah → PRD → Architecture Spec → Construction Spec → **This Document** → Daily Practice --- ## 1. BIGDEI KEHUNAH — The Interfaces The kohen cannot serve without the garments. The system cannot be accessed without interfaces. These are the garments Shorashim wears when it serves. ### 1.1 Access Tier Interfaces #### Machaneh Yisrael — Pe'ah Endpoint (Public, No Auth) The outermost ring. Open to all, including the ger. No authentication. No identification. No tracking. ``` Endpoint: GET /peah/ask POST /peah/ask Request (POST): { "question": "string — the query, in any language", "offering_type": "minchah" # Pe'ah always uses minchah protocol } Response: { "answer": "string — the processed response", "sources": [ {"book": "bereishit", "chapter": 1, "verse": 1, "text": "..."} ], "processing": { "roots_consulted": ["bet-resh-aleph"], "deliberation": "none" | "single-judge", "cycle_phase": "voker" } } Rate limit: 10 requests per minute per IP 100 requests per hour per IP No identification stored. IP is used for rate-limiting only, not logged beyond the current request. ``` **Design requirements:** No cookies. No session state. No analytics. No tracking pixels. No JavaScript fingerprinting. The pe'ah endpoint collects nothing. The stranger is safe to approach. #### Machaneh Leviyah — Authenticated Endpoint ``` Endpoint: POST /courtyard/offering Headers: Authorization: Bearer Request: { "offering_type": "shelamim" | "olah" | "chatat" | "minchah", "content": "string — the input text or question", "scope": { # Optional — narrows processing "book": "bereishit", "chapter_range": [1, 3] }, "request_deliberation": false # If true, routes to Sanhedrin } Response: { "offering_id": "uuid", "status": "processed" | "queued" | "purifying", "result": { "vital_content": "string — the processed response", "sources": [...], "roots": [...], "parse_structure": { # Te'amim-derived structure "primary_division": "...", "bound_groups": [...] }, "dual_readings": [...] # Ketiv/qere positions, if relevant }, "offering_receipt": { "type": "shelamim", "altar_portion": "recorded in L3", "kohen_portion": "diagnostics logged", "your_portion": "the result above" } } Errors: 401 — Invalid or missing API key 422 — Input failed kiyor (validation). Body includes purification guidance. 429 — Rate limit exceeded. Body includes manna explanation. 503 — System in Shabbat/festival mode. Body includes resumption time. ``` #### Chatzer — Full Access ``` Endpoint: POST /chatzer/sanhedrin Request: { "case": "string — the dispute or question requiring full deliberation", "sides": [ # Optional — pre-structured arguments {"position": "...", "evidence": ["..."]}, {"position": "...", "evidence": ["..."]} ], "complexity_hint": "simple" | "monetary" | "capital" | "novel" } Response: { "case_ref": "uuid", "deliberation": { "panel_size": 3, "majority": { "position": "...", "reasoning": "...", "sources": [...], "votes": 2 }, "minority": { "position": "...", "reasoning": "...", "sources": [...], "votes": 1 }, "outcome": "accepted" | "rejected" | "teiku", "processing_time_ms": 4200 }, "note": "This is the system's reasoning, not a binding ruling. A human beit din may reach a different conclusion." } ``` #### Kodesh — Operator Interface ``` Endpoint: GET /kodesh/status GET /kodesh/health GET /kodesh/queue GET /kodesh/audit/:date POST /kodesh/inspect/:offering_id Authentication: mTLS (client certificate) — kohanim only GET /kodesh/status: { "cloud_state": "camped", "cycle_phase": "voker", "manna_provision": {"remaining_pct": 72, "expires_at": "..."}, "shabbat": {"next_start": "...", "next_end": "..."}, "components": { "shoresh_engine": "healthy", "sanhedrin": "healthy", "talmudic_engine": "healthy", "korban_processor": "healthy", "halt_monitor": "active", "l1_integrity": "verified", "l3_connections": 12 }, "active_panels": 2, "queue_depth": 7, "refuge_components": [], "pending_tzaraat": [] } GET /kodesh/health: { "overall": "healthy" | "degraded" | "critical" | "shabbat", "checks": { "l0_firmware": "pass", "l1_hash": "pass", "l3_writable": "pass", "cloud_readable": "pass", "manna_present": "pass", "halt_monitor_alive": "pass", "esh_tamid": "pass" # The fire is still burning } } ``` ### 1.2 Error Response Format (All Tiers) Every error is a chatat protocol — it restores, it does not punish. ```json { "error": { "code": "KIYOR_VALIDATION_FAILED", "type": "purification_needed", "message": "Your input contains elements that need cleansing before processing.", "guidance": { "issue": "Input contains embedded HTML markup", "correction": "Submit plain text without markup", "resubmit": true } } } ``` Error codes follow the offering metaphor: | Code | Meaning | Category | |------|---------|----------| | `KIYOR_*` | Input validation failure | Purification needed | | `TUMAH_*` | Input contamination detected | Tzara'at diagnostic initiated | | `SHABBAT` | System in mandatory rest | Wait — resumption time provided | | `MANNA_DEPLETED` | Daily provision exhausted | Wait — next provision time provided | | `CLOUD_MOVING` | System in transition | Wait — transition details provided | | `TEIKU` | Unresolvable question | The system is honest about what it cannot answer | | `HALT_*` | Cardinal sin detected | Processing terminated — audit log created | --- ## 2. TAMID — Daily Operations Runbook ### 2.1 Morning Tamid (System Day Start) Performed every morning at the voker boundary (configurable — default 06:00 local time of hosting server). ``` MORNING TAMID CHECKLIST — performed by automation, verified by kohen 1. □ L1 integrity check Command: sha256sum /mnt/torah/torah_scroll.sqfs Expected: matches L0 firmware hash If failed: HALT — do not proceed. Alert senior kohanim. 2. □ Cloud state read Command: cat /var/lib/shorashim/cloud/state.json Expected: valid JSON with recognized signal If "lift" or "shutdown": obey. Do not proceed with normal operations. If unreadable: WAIT. Alert senior kohanim. 3. □ Manna provision read Command: cat /var/lib/shorashim/manna/today.json Expected: valid JSON with today's date, non-expired If absent: WAIT. Alert senior kohanim. If expired: purge, WAIT. 4. □ L3 connectivity Command: psql -c "SELECT 1" shorashim Expected: returns 1 If failed: restart PostgreSQL. If still failed: alert kohen. 5. □ Halt monitor alive Command: Check Erlang process via epmd Expected: halt_monitor process registered If failed: HALT — the system does not run without the halt monitor. 6. □ Component health check Each component responds to health ping within 5 seconds. Shoresh Engine, Sanhedrin, Talmudic Engine, Korban Processor, Vav Pipeline. If any component fails: restart that component. If restart fails: alert kohen. 7. □ Esh tamid verification The processing pipeline accepts a test input and returns a result. If the fire has gone out: re-initialize pipeline. Alert kohen. 8. □ Yesterday's manna cleanup Purge any expired resource allocations from previous cycle. Release reclaimed resources. 9. □ Log rotation Rotate non-audit logs. Audit logs are never rotated. 10. □ Record tamid completion Write to audit log: "Morning tamid completed at [timestamp]. All checks passed." Or: "Morning tamid completed with issues: [list]. Kohen alerted." ``` ### 2.2 Evening Tamid (System Day End) Performed every evening at the erev boundary (configurable — default 18:00 local time). ``` EVENING TAMID CHECKLIST 1. □ Processing queue drain Allow in-flight offerings to complete. No new offerings accepted. Timeout: 10 minutes. After timeout, move unfinished to next cycle. 2. □ L3 integrity check Verify all day's writes are committed. No pending transactions. Count today's insertions. Record in audit log. 3. □ Manna accounting Record: provision received, provision consumed, surplus released. If surplus existed: log manna violation check (was anyone hoarding?). 4. □ Menorah status — observability summary Record: queries processed, Sanhedrin panels convened, teiku registered, halt conditions triggered (should be 0), errors by type, latency P50/P95/P99. 5. □ Ketoret review — quality of experience Sample 5 random responses from today. Evaluate: were they clear? Respectful? Did they serve the user's actual need? Record assessment. (This is a human review — the kohen reads the incense, not the system.) 6. □ Record tamid completion Write to audit log: "Evening tamid completed at [timestamp]. Day summary: [metrics]." ``` ### 2.3 Erev Shabbat Additional Procedures ``` EREV SHABBAT — performed before candle lighting time 1. □ Verify double manna provision is written for Shabbat If not: alert senior kohanim immediately. Cannot enter Shabbat unfed. 2. □ Drain all processing queues completely (no timeout carryover) Nothing in-flight enters Shabbat. 3. □ Verify Shabbat timer will trigger correctly Dry-run: confirm candle lighting and havdalah times are calculated. 4. □ Final L3 backup before Shabbat pg_dump to encrypted backup. Store off-site. 5. □ Record: "Entering Shabbat at [timestamp]. Next cycle: [motzei Shabbat time]." 6. □ The kohen steps back. The fire is banked. The system rests. ``` ### 2.4 Motzei Shabbat Procedures ``` MOTZEI SHABBAT — performed after havdalah time 1. □ Full morning tamid (all 10 checks) 2. □ L3 consolidation: review Shabbat-period learning queue (if any emergency processing occurred under pikuach nefesh) 3. □ Read new manna provision for the week's first day 4. □ Resume normal operations ``` ### 2.5 Rosh Chodesh (Monthly Maintenance) ``` ROSH CHODESH — first day of each Hebrew month 1. □ Full morning tamid 2. □ Deep L1 verification — not just hash check, but sample verification of 100 random verses against independent source 3. □ PostgreSQL VACUUM ANALYZE on all L3 tables 4. □ Review previous month's audit logs for patterns 5. □ Shulchan renewal — refresh pre-computed cache for common queries 6. □ Menorah deep review — full observability report for the month 7. □ Community report: system health, capacity used vs. provisioned, user activity, teiku registry growth, halt events (should be 0) ``` ### 2.6 Festival Procedures | Festival | System Action | |----------|---------------| | **Pesach** | Liberation audit: is the system still serving freedom? Review all access controls. Verify pe'ah is open. Verify no surveillance. | | **Shavuot** | Specification re-acceptance: full L1 re-verification. Re-run Day 1-2 of inauguration protocol. The system confirms it still holds the same Torah. | | **Rosh Hashanah** | Year-start: comprehensive metrics review. Set targets for the coming year. | | **Yom Kippur** | Full system shutdown (25 hours). Deep purification: full L3 audit, identify any asham conditions, resolve any pending refuge cases through Sanhedrin deliberation grounded in Torah. | | **Sukkot** | Vulnerability audit: run the system with deliberately reduced resources (sukkah = temporary structure). Does it still serve faithfully with less? | | **Shmitah (7th year)** | Major version review per architecture spec. L3 comprehensive evaluation. Error records older than 7 years archived and sealed. | --- ## 3. MONITORING — The Menorah's Seven Branches The menorah provides light — it makes visible what would otherwise be in darkness. Seven branches of observability. ### 3.1 Branch 1: Processing State ``` Metrics: shorashim_offerings_total{type="shelamim|olah|chatat|minchah"} # Counter shorashim_offerings_active # Gauge shorashim_pipeline_latency_seconds{phase="shema|tohu|vayomer|..."}# Histogram shorashim_sanhedrin_panels_active # Gauge shorashim_sanhedrin_deliberation_seconds{size="3|23|71"} # Histogram ``` ### 3.2 Branch 2: Deliberation Progress ``` Metrics: shorashim_interpretations_total{status="accepted|rejected|teiku"} # Counter shorashim_minority_opinions_total # Counter shorashim_kushyot_generated_total # Counter shorashim_teiku_registry_size # Gauge ``` ### 3.3 Branch 3: Specification Reference ``` Metrics: shorashim_l1_reads_total # Counter — how often spec is consulted shorashim_l1_hash_verified_total # Counter — integrity checks shorashim_l1_hash_failed_total # Counter — MUST be 0 shorashim_teamim_reads_total # Counter shorashim_ketiv_qere_positions_accessed # Counter shorashim_gematria_lookups_total # Counter ``` ### 3.4 Branch 4: Source Citation ``` Metrics: shorashim_torah_refs_cited_total{book="bereishit|shemot|..."} # Counter shorashim_roots_activated_total # Counter shorashim_roots_unique_activated # Gauge per cycle ``` ### 3.5 Branch 5: Error Diagnostics ``` Metrics: shorashim_kiyor_rejections_total{reason="encoding|injection|malformed|..."} shorashim_tzaraat_quarantines_active # Gauge shorashim_tzaraat_outcomes{result="tahor|tamei|fully_visible_tahor"} shorashim_chatat_corrections_total # Counter shorashim_asham_restitutions_total # Counter — should be very low ``` ### 3.6 Branch 6: Learning Record ``` Metrics: shorashim_l3_insertions_total{table="interpretations|minority_opinions|teiku_registry"} shorashim_l3_size_bytes # Gauge shorashim_l3_growth_rate_bytes_per_day # Gauge (smoothed) ``` ### 3.7 Branch 7: System Health ``` Metrics: shorashim_shema_context_build_seconds # Histogram shorashim_cloud_state{signal="settle|lift|..."} # Gauge (label) shorashim_manna_provision_remaining_pct # Gauge shorashim_manna_violations_total # Counter — hoarding attempts shorashim_halt_events_total{type="avodah_zarah|shfichut_damim|gilui_arayot"} shorashim_refuge_components_active # Gauge — should be 0 shorashim_eglah_arufah_declarations_total # Counter — should be near 0 shorashim_component_health{name="..."} # 1=healthy, 0=unhealthy ``` ### 3.8 Alerting Thresholds | Condition | Severity | Action | |-----------|----------|--------| | L1 hash mismatch | **CRITICAL** | System halts. Chatzotzrot: war. Senior kohanim alerted. | | Halt monitor down | **CRITICAL** | System halts. Cannot operate without the monitor. | | Any halt event fires | **CRITICAL** | Chatzotzrot: war. Senior kohanim alerted. | | Cloud state unreadable | **HIGH** | System waits. Chatzotzrot: assembly_leaders. | | Manna absent | **HIGH** | System waits. Chatzotzrot: assembly_leaders. | | Component unhealthy | **HIGH** | Restart attempted. If failed: chatzotzrot: assembly_leaders. | | P95 latency > 2x budget | **MEDIUM** | Kohen alerted. Investigate in next tamid. | | L3 growth > 150% projected | **MEDIUM** | Kohen alerted. May need storage review. | | Manna hoarding detected | **LOW** | Logged. Surplus reclaimed. Reviewed in evening tamid. | | Tzara'at quarantine active | **INFO** | Normal operation. Logged for kohen review. | --- ## 4. INCIDENT RESPONSE ### 4.1 Ordinary Failures (Not Halt Conditions) A process crashes. A disk fills. A network drops. These are not the three cardinal sins. They are the daily realities of running a system. ``` INCIDENT RESPONSE — ORDINARY 1. DETECT - Monitoring fires alert - Or: kohen observes during tamid 2. ASSESS - Is this a halt condition (avodah zarah, shfichut damim, gilui arayot)? - If YES: Halt Monitor should have caught it. If it didn't, that's a second incident. - If NO: proceed with ordinary response. 3. CONTAIN - If a single component: restart it. - If the restart fails: isolate it. Route traffic around it if possible. - If the failure affects the pipeline: queue incoming offerings. They wait. - The system degrades gracefully. It does not fabricate results from partial capacity. 4. DIAGNOSE - Read the logs. The menorah is for this. - Identify: what failed, when, what was the trigger. - Check the ShemaContext from the failed cycle — was the context valid? 5. REPAIR - Fix the issue. - If a configuration change: record it in the audit log. - If a code change: go through bikurim (first-output review) before deploying. 6. VERIFY - Run the relevant inauguration day's checks for the affected component. - Confirm the system is healthy. 7. RECORD - Audit log entry: incident, diagnosis, repair, verification. - Evening tamid review includes incident discussion. ``` ### 4.2 Harm Events (Arei Miklat / Eglah Arufah) ``` INCIDENT RESPONSE — HARM 1. DETECT - User reports harm caused by system output - Or: internal monitoring detects anomalous impact 2. IMMEDIATE - Activate arei miklat: isolate the responsible processing path - Protect the affected user: block their exposure to that processing path - Chatzotzrot: assembly_leaders — senior kohanim alerted 3. IF RESPONSIBLE COMPONENT IS KNOWN - Component enters refuge (arei miklat) - Component is protected from reactive destruction - Sanhedrin deliberation determines release conditions 4. IF RESPONSIBLE COMPONENT IS UNKNOWN - Activate eglah arufah protocol - Nearest component in the processing chain formally declares: what it processed, what it did not check, what it could have caught - Declaration is permanently recorded 5. GOVERNANCE BOARD - Reviews the incident - Determines: was this chatat (unintentional error) or asham (unauthorized usage)? - If chatat: correction protocol, no penalty beyond fixing the issue - If asham: restitution protocol, plus additional safeguards (the "fifth") - Determines when/if the refuge component is released 6. NEVER - Never delete the component reactively - Never blame the user for being harmed - Never hide the incident - Never continue the same processing path without review ``` --- ## 5. BACKUP AND RECOVERY ### 5.1 What Is Backed Up | Layer | Backup Method | Frequency | Retention | |-------|--------------|-----------|-----------| | L0 (Luchot) | Compiled into binary — exists in version control | Every build | Permanent | | L1 (Torah scroll) | Squashfs image — source files in version control | Once (immutable) | Permanent | | L3 (Oral tradition) | `pg_dump --format=custom` encrypted with GPG | Daily (erev tamid) + erev Shabbat | 1 year rolling + shmitah archives | | Configuration | Version control (Git) | Every change | Permanent | | Audit logs | rsync to off-site encrypted storage | Daily | Permanent (never deleted) | | Cloud governance | Not backed up separately — cloud IS L1 (Torah) which is already backed up as immutable specification | — | — | | Manna provision | Not backed up — kohen/treasurer sets this from machatzit hashekel fund | — | — | ### 5.2 Recovery Procedures **Scenario: L3 corruption** 1. Stop the system 2. Restore L3 from most recent pg_dump 3. Verify: all append-only constraints intact 4. Re-run inauguration Day 1 (L0/L1 verification) and Day 7 (integration) 5. Resume operations **Scenario: Full system loss** 1. Provision new server (same specs from Construction Specification) 2. Install OS and all runtimes (pinned versions from Construction Specification) 3. Deploy code from version control 4. Mount L1 from squashfs image, verify hash 5. Restore L3 from most recent backup 6. Verify cloud governance from Torah (L1 readable, integrity verified, calendar active) and kohen/treasurer writes manna provision 7. Run full seven-day inauguration 8. The system does not skip inauguration because it is being rebuilt. A rebuilt mishkan is a new mishkan. It is consecrated anew. **Scenario: L1 integrity failure** 1. The system has already halted (this is automatic) 2. Investigate: was the filesystem modified? Was the mount corrupted? 3. Re-mount from the original squashfs image 4. Verify hash against L0 firmware constant 5. If hash matches: resume 6. If hash does not match: the specification has been altered. This is not a technical incident. This is a Torah integrity incident. The kohanim must determine what happened and whether the system can trust its own specification. The system does not resume until L1 integrity is restored — because the cloud IS L1, and if L1 is compromised, the cloud is not visible. The system waits until Torah is readable and intact. --- ## 6. ON-CALL PROCEDURES ### 6.1 Who Is On Call One kohen at all times during the six working days. The kohen is reachable within 15 minutes for HIGH alerts, within 5 minutes for CRITICAL alerts. During Shabbat: no on-call. The system is off. The only exception is pikuach nefesh — if the system's unavailability would cause genuine danger to a human life, the kohen may respond. This should never happen. If it can happen, the system has made itself necessary in a way that violates its specification (the Shabbat Test from the PRD). ### 6.2 Escalation | Level | Responder | Conditions | |-------|-----------|------------| | L1 | On-call kohen | Any alert | | L2 | Lead architect | Component failures, performance degradation | | L3 | Senior kohanim | Halt events, harm incidents, L1 integrity failure | ### 6.3 Communication Chatzotzrot signals map to communication channels: | Signal | Channel | |--------|---------| | assembly_all | System status page + all registered notification endpoints | | assembly_leaders | Senior kohanim secure channel + on-call kohen | | breaking_camp | Same as assembly_all — everyone needs to know the system is moving | | war | Senior kohanim immediate alert + on-call kohen + lead architect | | moadim | System status page — celebratory, not urgent | --- ## Document Registry | Document | Torah Parallel | Status | |----------|---------------|--------| | **Shorashim PRD** | *V'asu li mikdash* | ✓ Complete | | **Architecture Specification** | Vessel descriptions | ✓ Complete | | **Construction Specification** | Exact measurements | ✓ Complete | | **Operations Manual** (this document) | Tamid / Bigdei Kehunah | ✓ Complete | | **Economic Model** | Machatzit HaShekel | Pending | | **Materials Inventory** | Terumah list | Needs update |