L1 data integrity: preserve the edut whole — Shema, middle letter, ketiv/qere, smikhut

Shemot 25:16 — v'natata el ha-aron et ha-edut asher eten elekha. The edut placed in the aron must be complete. The Masoretic tradition preserves every letter, every enlarged letter, every ketiv/qere pair — because each carries meaning the scribe cannot reconstruct. The previous build_l1.py silently dropped: - enlarged letters encoded as <seg type="x-large"> nested in <w> → the Shema was corrupt: שמ instead of שמע, אח instead of אחד → the middle letter of Torah was corrupt: גח instead of גחון → the Zelophchad nun was missing: משפט instead of משפטן - ketiv/qere dual readings encoded as <note type="variant">/<rdg type="x-qere"> → the spec committed to preserving both; the code returned [] - smikhut (construct state) pairs marked in morphology codes → the spec committed to extracting them; the code returned [] Fixes: - collect_word_text() walks all nested <seg> children of <w> and records large_letters / small_letters metadata with character offsets. - parse_verse() recognizes <w type="x-ketiv"> followed by <note>, extracts both the ketiv and the qere readings, stores a ketiv_qere list on the verse. - _extract_smikhut() scans words for construct-state morph codes (ending in "c" on N/A primary segments, excluding pronominal suffixes) and records construct/absolute pairs. - L1 SHA-256 hash updated everywhere: integrity.sha256, l1_reader.ex, shoresh-engine's Luchot.Firmware, its test, KOHEN.md. Torah-wide results after fix: - 4 enlarged letters preserved (was 0): Shema ayin, Shema dalet, gachon vav (middle of Torah), Zelophchad nun - 67 ketiv/qere pairs captured (was 0) - 9,203 smikhut pairs extracted (was 0) Elixir updates: - l1_reader.ex: normalize_verse now surfaces ketiv_qere, smikhut, and per-word large_letters/small_letters. New type specs. New extract_large_letters/1 helper. extract_ketiv_qere and extract_smikhut now return real data instead of []. - step31_shema_test.exs: replaced placeholder asserts with real integrity checks. Four new tests verify the Shema word is complete (שמע not שמ), echad is complete (אחד not אח), the enlarged letters are marked, the middle letter of Torah is preserved, and ketiv/qere pairs from Bereishit 8:17 are both captured. Build script: - build_l1.py no longer rmtrees the entire l1/ directory (preserves l1/README.md). Only regenerates torah/ + build_report.json + integrity.sha256. - Accepts SHORASHIM_SOURCE_DATA env var or CLI arg for morphhb path. Test counts: - Elixir: 260 (was 254), 0 failures - Haskell: 33, 0 failures - Prolog: 195, 0 failures The system that can hear שמע correctly has passed its own shema test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-21 13:11:52 -04:00
1 parent 04713cc884
commit d4d0a6fcb6
@@ -83,5 +83,14 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 6,
+      "construct": "אִמְרֵי",
+      "construct_consonants": "אמרי",
+      "absolute": "פִֽי",
+      "absolute_consonants": "פי"
+    }
  ]
 }
@@ -128,5 +128,21 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "בְּאֶ֣רֶץ",
+      "construct_consonants": "בארץ",
+      "absolute": "מִדְבָּ֔ר",
+      "absolute_consonants": "מדבר"
+    },
+    {
+      "position": 10,
+      "construct": "כְּאִישׁ֥וֹן",
+      "construct_consonants": "כאישון",
+      "absolute": "עֵינֽוֹ",
+      "absolute_consonants": "עינו"
+    }
  ]
 }
@@ -80,5 +80,14 @@
      ],
      "shoresh": "נכר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 6,
+      "construct": "אֵ֥ל",
+      "construct_consonants": "אל",
+      "absolute": "נֵכָֽר",
+      "absolute_consonants": "נכר"
+    }
  ]
 }
@@ -148,5 +148,37 @@
      ],
      "shoresh": "צור"
    }
+  ],
+  "ketiv_qere": [
+    {
+      "word_position": 3,
+      "ketiv": "במותי",
+      "ketiv_consonants": "במותי",
+      "qere": "בָּ֣מֳתֵי",
+      "qere_consonants": "במתי"
+    }
+  ],
+  "smikhut": [
+    {
+      "position": 3,
+      "construct": "במותי",
+      "construct_consonants": "במותי",
+      "absolute": "אָ֔רֶץ",
+      "absolute_consonants": "ארץ"
+    },
+    {
+      "position": 6,
+      "construct": "תְּנוּבֹ֣ת",
+      "construct_consonants": "תנובת",
+      "absolute": "שָׂדָ֑י",
+      "absolute_consonants": "שדי"
+    },
+    {
+      "position": 12,
+      "construct": "מֵחַלְמִ֥ישׁ",
+      "construct_consonants": "מחלמיש",
+      "absolute": "צֽוּר",
+      "absolute_consonants": "צור"
+    }
  ]
 }
@@ -208,5 +208,56 @@
      ],
      "shoresh": "חמר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 1,
+      "construct": "חֶמְאַ֨ת",
+      "construct_consonants": "חמאת",
+      "absolute": "בָּקָ֜ר",
+      "absolute_consonants": "בקר"
+    },
+    {
+      "position": 3,
+      "construct": "וַחֲלֵ֣ב",
+      "construct_consonants": "וחלב",
+      "absolute": "צֹ֗אן",
+      "absolute_consonants": "צאן"
+    },
+    {
+      "position": 6,
+      "construct": "חֵ֨לֶב",
+      "construct_consonants": "חלב",
+      "absolute": "כָּרִ֜ים",
+      "absolute_consonants": "כרים"
+    },
+    {
+      "position": 9,
+      "construct": "בְּנֵֽי",
+      "construct_consonants": "בני",
+      "absolute": "בָשָׁן֙",
+      "absolute_consonants": "בשן"
+    },
+    {
+      "position": 13,
+      "construct": "חֵ֖לֶב",
+      "construct_consonants": "חלב",
+      "absolute": "כִּלְי֣וֹת",
+      "absolute_consonants": "כליות"
+    },
+    {
+      "position": 14,
+      "construct": "כִּלְי֣וֹת",
+      "construct_consonants": "כליות",
+      "absolute": "חִטָּ֑ה",
+      "absolute_consonants": "חטה"
+    },
+    {
+      "position": 16,
+      "construct": "וְדַם",
+      "construct_consonants": "ודם",
+      "absolute": "עֵנָ֖ב",
+      "absolute_consonants": "ענב"
+    }
  ]
 }
@@ -139,5 +139,14 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 11,
+      "construct": "צ֥וּר",
+      "construct_consonants": "צור",
+      "absolute": "יְשֻׁעָתֽוֹ",
+      "absolute_consonants": "ישעתו"
+    }
  ]
 }
@@ -73,5 +73,14 @@
      ],
      "prefix": "ו"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 4,
+      "construct": "מִכַּ֥עַס",
+      "construct_consonants": "מכעס",
+      "absolute": "בָּנָ֖יו",
+      "absolute_consonants": "בניו"
+    }
  ]
 }
@@ -161,5 +161,14 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 9,
+      "construct": "ד֤וֹר",
+      "construct_consonants": "דור",
+      "absolute": "תַּהְפֻּכֹת֙",
+      "absolute_consonants": "תהפכת"
+    }
  ]
 }
@@ -158,5 +158,14 @@
      ],
      "shoresh": "הר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 13,
+      "construct": "מוֹסְדֵ֥י",
+      "construct_consonants": "מוסדי",
+      "absolute": "הָרִֽים",
+      "absolute_consonants": "הרים"
+    }
  ]
 }
@@ -154,5 +154,28 @@
      ],
      "shoresh": "עפר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 1,
+      "construct": "מְזֵ֥י",
+      "construct_consonants": "מזי",
+      "absolute": "רָעָ֛ב",
+      "absolute_consonants": "רעב"
+    },
+    {
+      "position": 7,
+      "construct": "וְשֶׁן",
+      "construct_consonants": "ושן",
+      "absolute": "בְּהֵמוֹת֙",
+      "absolute_consonants": "בהמות"
+    },
+    {
+      "position": 12,
+      "construct": "חֲמַ֖ת",
+      "construct_consonants": "חמת",
+      "absolute": "זֹחֲלֵ֥י",
+      "absolute_consonants": "זחלי"
+    }
  ]
 }
@@ -139,5 +139,14 @@
      ],
      "shoresh": "שיב"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 12,
+      "construct": "אִ֥ישׁ",
+      "construct_consonants": "איש",
+      "absolute": "שֵׂיבָֽה",
+      "absolute_consonants": "שיבה"
+    }
  ]
 }
@@ -172,5 +172,21 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "כַּ֤עַס",
+      "construct_consonants": "כעס",
+      "absolute": "אוֹיֵב֙",
+      "absolute_consonants": "אויב"
+    },
+    {
+      "position": 15,
+      "construct": "כָּל",
+      "construct_consonants": "כל",
+      "absolute": "זֹֽאת",
+      "absolute_consonants": "זאת"
+    }
  ]
 }
@@ -81,5 +81,14 @@
      ],
      "prefix": "ל"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "שֵׁ֥ם",
+      "construct_consonants": "שם",
+      "absolute": "יְהוָ֖ה",
+      "absolute_consonants": "יהוה"
+    }
  ]
 }
@@ -134,5 +134,35 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "מִגֶּ֤פֶן",
+      "construct_consonants": "מגפן",
+      "absolute": "סְדֹם֙",
+      "absolute_consonants": "סדם"
+    },
+    {
+      "position": 5,
+      "construct": "וּמִשַּׁדְמֹ֖ת",
+      "construct_consonants": "ומשדמת",
+      "absolute": "עֲמֹרָ֑ה",
+      "absolute_consonants": "עמרה"
+    },
+    {
+      "position": 8,
+      "construct": "עִנְּבֵי",
+      "construct_consonants": "ענבי",
+      "absolute": "ר֔וֹשׁ",
+      "absolute_consonants": "רוש"
+    },
+    {
+      "position": 10,
+      "construct": "אַשְׁכְּלֹ֥ת",
+      "construct_consonants": "אשכלת",
+      "absolute": "מְרֹרֹ֖ת",
+      "absolute_consonants": "מררת"
+    }
  ]
 }
@@ -71,5 +71,21 @@
      ],
      "shoresh": "אכז"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 1,
+      "construct": "חֲמַ֥ת",
+      "construct_consonants": "חמת",
+      "absolute": "תַּנִּינִ֖ם",
+      "absolute_consonants": "תנינם"
+    },
+    {
+      "position": 4,
+      "construct": "וְרֹ֥אשׁ",
+      "construct_consonants": "וראש",
+      "absolute": "פְּתָנִ֖ים",
+      "absolute_consonants": "פתנים"
+    }
  ]
 }
@@ -146,5 +146,21 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 4,
+      "construct": "לְעֵ֖ת",
+      "construct_consonants": "לעת",
+      "absolute": "תָּמ֣וּט",
+      "absolute_consonants": "תמוט"
+    },
+    {
+      "position": 9,
+      "construct": "י֣וֹם",
+      "construct_consonants": "יום",
+      "absolute": "אֵידָ֔ם",
+      "absolute_consonants": "אידם"
+    }
  ]
 }
@@ -135,5 +135,21 @@
      ],
      "shoresh": "סתר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "חֵ֤לֶב",
+      "construct_consonants": "חלב",
+      "absolute": "זְבָחֵ֨ימוֹ֙",
+      "absolute_consonants": "זבחימו"
+    },
+    {
+      "position": 6,
+      "construct": "יֵ֣ין",
+      "construct_consonants": "יין",
+      "absolute": "נְסִיכָ֑ם",
+      "absolute_consonants": "נסיכם"
+    }
  ]
 }
@@ -155,5 +155,21 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "כָל",
+      "construct_consonants": "כל",
+      "absolute": "דְּרָכָ֖יו",
+      "absolute_consonants": "דרכיו"
+    },
+    {
+      "position": 8,
+      "construct": "אֵ֤ל",
+      "construct_consonants": "אל",
+      "absolute": "אֱמוּנָה֙",
+      "absolute_consonants": "אמונה"
+    }
  ]
 }
@@ -134,5 +134,14 @@
      ],
      "shoresh": "שלם"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 3,
+      "construct": "בְּרַ֣ק",
+      "construct_consonants": "ברק",
+      "absolute": "חַרְבִּ֔י",
+      "absolute_consonants": "חרבי"
+    }
  ]
 }
@@ -140,5 +140,28 @@
      ],
      "shoresh": "אוי"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 7,
+      "construct": "מִדַּ֤ם",
+      "construct_consonants": "מדם",
+      "absolute": "חָלָל֙",
+      "absolute_consonants": "חלל"
+    },
+    {
+      "position": 10,
+      "construct": "מֵרֹ֖אשׁ",
+      "construct_consonants": "מראש",
+      "absolute": "פַּרְע֥וֹת",
+      "absolute_consonants": "פרעות"
+    },
+    {
+      "position": 11,
+      "construct": "פַּרְע֥וֹת",
+      "construct_consonants": "פרעות",
+      "absolute": "אוֹיֵֽב",
+      "absolute_consonants": "אויב"
+    }
  ]
 }
@@ -143,5 +143,14 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "דַם",
+      "construct_consonants": "דם",
+      "absolute": "עֲבָדָ֖יו",
+      "absolute_consonants": "עבדיו"
+    }
  ]
 }
@@ -157,5 +157,35 @@
      ],
      "shoresh": "נון"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "כָּל",
+      "construct_consonants": "כל",
+      "absolute": "דִּבְרֵ֥י",
+      "absolute_consonants": "דברי"
+    },
+    {
+      "position": 6,
+      "construct": "דִּבְרֵ֥י",
+      "construct_consonants": "דברי",
+      "absolute": "הַשִּׁירָֽה",
+      "absolute_consonants": "השירה"
+    },
+    {
+      "position": 9,
+      "construct": "בְּאָזְנֵ֣י",
+      "construct_consonants": "באזני",
+      "absolute": "הָעָ֑ם",
+      "absolute_consonants": "העם"
+    },
+    {
+      "position": 13,
+      "construct": "בִּן",
+      "construct_consonants": "בן",
+      "absolute": "נֽוּן",
+      "absolute_consonants": "נון"
+    }
  ]
 }
@@ -108,5 +108,21 @@
      ],
      "shoresh": "ישר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "כָּל",
+      "construct_consonants": "כל",
+      "absolute": "הַדְּבָרִ֥ים",
+      "absolute_consonants": "הדברים"
+    },
+    {
+      "position": 9,
+      "construct": "כָּל",
+      "construct_consonants": "כל",
+      "absolute": "יִשְׂרָאֵֽל",
+      "absolute_consonants": "ישראל"
+    }
  ]
 }
@@ -236,5 +236,28 @@
      ],
      "prefix": "ה"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "לְכָל",
+      "construct_consonants": "לכל",
+      "absolute": "הַדְּבָרִ֔ים",
+      "absolute_consonants": "הדברים"
+    },
+    {
+      "position": 19,
+      "construct": "כָּל",
+      "construct_consonants": "כל",
+      "absolute": "דִּבְרֵ֖י",
+      "absolute_consonants": "דברי"
+    },
+    {
+      "position": 20,
+      "construct": "דִּבְרֵ֖י",
+      "construct_consonants": "דברי",
+      "absolute": "הַתּוֹרָ֥ה",
+      "absolute_consonants": "התורה"
+    }
  ]
 }
@@ -94,5 +94,14 @@
      "prefix": "ל",
      "shoresh": "אמר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "בְּעֶ֛צֶם",
+      "construct_consonants": "בעצם",
+      "absolute": "הַיּ֥וֹם",
+      "absolute_consonants": "היום"
+    }
  ]
 }
@@ -259,5 +259,49 @@
      "prefix": "ל",
      "shoresh": "אחז"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 3,
+      "construct": "הַר֩",
+      "construct_consonants": "הר",
+      "absolute": "הָעֲבָרִ֨ים",
+      "absolute_consonants": "העברים"
+    },
+    {
+      "position": 6,
+      "construct": "הַר",
+      "construct_consonants": "הר",
+      "absolute": "נְב֗וֹ",
+      "absolute_consonants": "נבו"
+    },
+    {
+      "position": 9,
+      "construct": "בְּאֶ֣רֶץ",
+      "construct_consonants": "בארץ",
+      "absolute": "מוֹאָ֔ב",
+      "absolute_consonants": "מואב"
+    },
+    {
+      "position": 13,
+      "construct": "פְּנֵ֣י",
+      "construct_consonants": "פני",
+      "absolute": "יְרֵח֑וֹ",
+      "absolute_consonants": "ירחו"
+    },
+    {
+      "position": 17,
+      "construct": "אֶ֣רֶץ",
+      "construct_consonants": "ארץ",
+      "absolute": "כְּנַ֔עַן",
+      "absolute_consonants": "כנען"
+    },
+    {
+      "position": 22,
+      "construct": "לִבְנֵ֥י",
+      "construct_consonants": "לבני",
+      "absolute": "יִשְׂרָאֵ֖ל",
+      "absolute_consonants": "ישראל"
+    }
  ]
 }
@@ -219,5 +219,56 @@
      ],
      "shoresh": "ישר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "בְּתוֹךְ֙",
+      "construct_consonants": "בתוך",
+      "absolute": "בְּנֵ֣י",
+      "absolute_consonants": "בני"
+    },
+    {
+      "position": 6,
+      "construct": "בְּנֵ֣י",
+      "construct_consonants": "בני",
+      "absolute": "יִשְׂרָאֵ֔ל",
+      "absolute_consonants": "ישראל"
+    },
+    {
+      "position": 8,
+      "construct": "בְּמֵֽי",
+      "construct_consonants": "במי",
+      "absolute": "מְרִיבַ֥ת",
+      "absolute_consonants": "מריבת"
+    },
+    {
+      "position": 9,
+      "construct": "מְרִיבַ֥ת",
+      "construct_consonants": "מריבת",
+      "absolute": "קָדֵ֖שׁ",
+      "absolute_consonants": "קדש"
+    },
+    {
+      "position": 11,
+      "construct": "מִדְבַּר",
+      "construct_consonants": "מדבר",
+      "absolute": "צִ֑ן",
+      "absolute_consonants": "צן"
+    },
+    {
+      "position": 18,
+      "construct": "בְּת֖וֹךְ",
+      "construct_consonants": "בתוך",
+      "absolute": "בְּנֵ֥י",
+      "absolute_consonants": "בני"
+    },
+    {
+      "position": 19,
+      "construct": "בְּנֵ֥י",
+      "construct_consonants": "בני",
+      "absolute": "יִשְׂרָאֵֽל",
+      "absolute_consonants": "ישראל"
+    }
  ]
 }
@@ -162,5 +162,14 @@
      ],
      "shoresh": "ישר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 14,
+      "construct": "לִבְנֵ֥י",
+      "construct_consonants": "לבני",
+      "absolute": "יִשְׂרָאֵֽל",
+      "absolute_consonants": "ישראל"
+    }
  ]
 }
@@ -146,5 +146,21 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "יְמ֣וֹת",
+      "construct_consonants": "ימות",
+      "absolute": "עוֹלָ֔ם",
+      "absolute_consonants": "עולם"
+    },
+    {
+      "position": 5,
+      "construct": "שְׁנ֣וֹת",
+      "construct_consonants": "שנות",
+      "absolute": "דּוֹר",
+      "absolute_consonants": "דור"
+    }
  ]
 }
@@ -139,5 +139,35 @@
      ],
      "shoresh": "ישר"
    }
+  ],
+  "smikhut": [
+    {
+      "position": 5,
+      "construct": "בְּנֵ֣י",
+      "construct_consonants": "בני",
+      "absolute": "אָדָ֑ם",
+      "absolute_consonants": "אדם"
+    },
+    {
+      "position": 8,
+      "construct": "גְּבֻלֹ֣ת",
+      "construct_consonants": "גבלת",
+      "absolute": "עַמִּ֔ים",
+      "absolute_consonants": "עמים"
+    },
+    {
+      "position": 10,
+      "construct": "לְמִסְפַּ֖ר",
+      "construct_consonants": "למספר",
+      "absolute": "בְּנֵ֥י",
+      "absolute_consonants": "בני"
+    },
+    {
+      "position": 11,
+      "construct": "בְּנֵ֥י",
+      "construct_consonants": "בני",
+      "absolute": "יִשְׂרָאֵֽל",
+      "absolute_consonants": "ישראל"
+    }
  ]
 }
@@ -79,5 +79,21 @@
        "meteg_siluq"
      ]
    }
+  ],
+  "smikhut": [
+    {
+      "position": 2,
+      "construct": "חֵ֥לֶק",
+      "construct_consonants": "חלק",
+      "absolute": "יְהֹוָ֖ה",
+      "absolute_consonants": "יהוה"
+    },
+    {
+      "position": 6,
+      "construct": "חֶ֥בֶל",
+      "construct_consonants": "חבל",
+      "absolute": "נַחֲלָתֽוֹ",
+      "absolute_consonants": "נחלתו"
+    }
  ]
 }