L1 data integrity: preserve the edut whole — Shema, middle letter, ketiv/qere, smikhut

Shemot 25:16 — v'natata el ha-aron et ha-edut asher eten elekha.
The edut placed in the aron must be complete. The Masoretic
tradition preserves every letter, every enlarged letter, every
ketiv/qere pair — because each carries meaning the scribe cannot
reconstruct.

The previous build_l1.py silently dropped:
- enlarged letters encoded as <seg type="x-large"> nested in <w>
  → the Shema was corrupt: שמ instead of שמע, אח instead of אחד
  → the middle letter of Torah was corrupt: גח instead of גחון
  → the Zelophchad nun was missing: משפט instead of משפטן
- ketiv/qere dual readings encoded as <note type="variant">/<rdg type="x-qere">
  → the spec committed to preserving both; the code returned []
- smikhut (construct state) pairs marked in morphology codes
  → the spec committed to extracting them; the code returned []

Fixes:
- collect_word_text() walks all nested <seg> children of <w> and records
  large_letters / small_letters metadata with character offsets.
- parse_verse() recognizes <w type="x-ketiv"> followed by <note>, extracts
  both the ketiv and the qere readings, stores a ketiv_qere list on the verse.
- _extract_smikhut() scans words for construct-state morph codes
  (ending in "c" on N/A primary segments, excluding pronominal suffixes)
  and records construct/absolute pairs.
- L1 SHA-256 hash updated everywhere: integrity.sha256, l1_reader.ex,
  shoresh-engine's Luchot.Firmware, its test, KOHEN.md.

Torah-wide results after fix:
- 4 enlarged letters preserved (was 0): Shema ayin, Shema dalet,
  gachon vav (middle of Torah), Zelophchad nun
- 67 ketiv/qere pairs captured (was 0)
- 9,203 smikhut pairs extracted (was 0)

Elixir updates:
- l1_reader.ex: normalize_verse now surfaces ketiv_qere, smikhut,
  and per-word large_letters/small_letters. New type specs. New
  extract_large_letters/1 helper. extract_ketiv_qere and
  extract_smikhut now return real data instead of [].
- step31_shema_test.exs: replaced placeholder asserts with real
  integrity checks. Four new tests verify the Shema word is complete
  (שמע not שמ), echad is complete (אחד not אח), the enlarged
  letters are marked, the middle letter of Torah is preserved,
  and ketiv/qere pairs from Bereishit 8:17 are both captured.

Build script:
- build_l1.py no longer rmtrees the entire l1/ directory (preserves
  l1/README.md). Only regenerates torah/ + build_report.json +
  integrity.sha256.
- Accepts SHORASHIM_SOURCE_DATA env var or CLI arg for morphhb path.

Test counts:
- Elixir: 260 (was 254), 0 failures
- Haskell: 33, 0 failures
- Prolog: 195, 0 failures

The system that can hear שמע correctly has passed its own shema test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Opus 4.6 committed 2026-04-21 13:11:52 -04:00
1 parent 04713cc884
commit d4d0a6fcb6
4137 files changed
+73656 -85

No files matched your search

+9
View File
@@ -82,5 +82,14 @@
"prefix": "ל",
"shoresh": "אמר"
}
],
"smikhut": [
{
"position": 5,
"construct": "בְּהַ֥ר",
"construct_consonants": "בהר",
"absolute": "סִינַ֖י",
"absolute_consonants": "סיני"
}
]
}
+16
View File
@@ -240,5 +240,21 @@
],
"shoresh": "שבו"
}
],
"smikhut": [
{
"position": 3,
"construct": "שְׁנַ֤ת",
"construct_consonants": "שנת",
"absolute": "הַחֲמִשִּׁים֙",
"absolute_consonants": "החמשים"
},
{
"position": 9,
"construct": "לְכָל",
"construct_consonants": "לכל",
"absolute": "יֹשְׁבֶ֑יהָ",
"absolute_consonants": "ישביה"
}
]
}
+9
View File
@@ -182,5 +182,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 3,
"construct": "שְׁנַ֛ת",
"construct_consonants": "שנת",
"absolute": "הַחֲמִשִּׁ֥ים",
"absolute_consonants": "החמשים"
}
]
}
+9
View File
@@ -80,5 +80,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 1,
"construct": "בִּשְׁנַ֥ת",
"construct_consonants": "בשנת",
"absolute": "הַיּוֹבֵ֖ל",
"absolute_consonants": "היובל"
}
]
}
+9
View File
@@ -140,5 +140,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 7,
"construct": "מִיַּ֣ד",
"construct_consonants": "מיד",
"absolute": "עֲמִיתֶ֑ךָ",
"absolute_consonants": "עמיתך"
}
]
}
+23
View File
@@ -135,5 +135,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 1,
"construct": "בְּמִסְפַּ֤ר",
"construct_consonants": "במספר",
"absolute": "שָׁנִים֙",
"absolute_consonants": "שנים"
},
{
"position": 8,
"construct": "בְּמִסְפַּ֥ר",
"construct_consonants": "במספר",
"absolute": "שְׁנֵֽי",
"absolute_consonants": "שני"
},
{
"position": 9,
"construct": "שְׁנֵֽי",
"construct_consonants": "שני",
"absolute": "תְבוּאֹ֖ת",
"absolute_consonants": "תבואת"
}
]
}
+30
View File
@@ -180,5 +180,35 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 1,
"construct": "לְפִ֣י",
"construct_consonants": "לפי",
"absolute": "רֹ֣ב",
"absolute_consonants": "רב"
},
{
"position": 2,
"construct": "רֹ֣ב",
"construct_consonants": "רב",
"absolute": "הַשָּׁנִ֗ים",
"absolute_consonants": "השנים"
},
{
"position": 6,
"construct": "וּלְפִי֙",
"construct_consonants": "ולפי",
"absolute": "מְעֹ֣ט",
"absolute_consonants": "מעט"
},
{
"position": 12,
"construct": "מִסְפַּ֣ר",
"construct_consonants": "מספר",
"absolute": "תְּבוּאֹ֔ת",
"absolute_consonants": "תבואת"
}
]
}
+9
View File
@@ -198,5 +198,14 @@
"prefix": "ל",
"shoresh": "יהו"
}
],
"smikhut": [
{
"position": 3,
"construct": "בְּנֵ֤י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵל֙",
"absolute_consonants": "ישראל"
}
]
}
+16
View File
@@ -72,5 +72,21 @@
"prefix": "ל",
"shoresh": "ארץ"
}
],
"smikhut": [
{
"position": 1,
"construct": "וּבְכֹ֖ל",
"construct_consonants": "ובכל",
"absolute": "אֶ֣רֶץ",
"absolute_consonants": "ארץ"
},
{
"position": 2,
"construct": "אֶ֣רֶץ",
"construct_consonants": "ארץ",
"absolute": "אֲחֻזַּתְכֶ֑ם",
"absolute_consonants": "אחזתכם"
}
]
}
+9
View File
@@ -147,5 +147,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 12,
"construct": "מִמְכַּ֥ר",
"construct_consonants": "ממכר",
"absolute": "אָחִֽיו",
"absolute_consonants": "אחיו"
}
]
}
+9
View File
@@ -125,5 +125,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 10,
"construct": "כְּדֵ֥י",
"construct_consonants": "כדי",
"absolute": "גְאֻלָּתֽוֹ",
"absolute_consonants": "גאלתו"
}
]
}
+9
View File
@@ -145,5 +145,14 @@
],
"prefix": "ל"
}
],
"smikhut": [
{
"position": 3,
"construct": "שְׁנֵ֣י",
"construct_consonants": "שני",
"absolute": "מִמְכָּר֔וֹ",
"absolute_consonants": "ממכרו"
}
]
}
+23
View File
@@ -216,5 +216,28 @@
],
"prefix": "ל"
}
],
"smikhut": [
{
"position": 5,
"construct": "דֵּי֮",
"construct_consonants": "די",
"absolute": "הָשִׁ֣יב",
"absolute_consonants": "השיב"
},
{
"position": 10,
"construct": "בְּיַד֙",
"construct_consonants": "ביד",
"absolute": "הַקֹּנֶ֣ה",
"absolute_consonants": "הקנה"
},
{
"position": 14,
"construct": "שְׁנַ֣ת",
"construct_consonants": "שנת",
"absolute": "הַיּוֹבֵ֑ל",
"absolute_consonants": "היובל"
}
]
}
+23
View File
@@ -174,5 +174,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 4,
"construct": "בֵּית",
"construct_consonants": "בית",
"absolute": "מוֹשַׁב֙",
"absolute_consonants": "מושב"
},
{
"position": 5,
"construct": "מוֹשַׁב֙",
"construct_consonants": "מושב",
"absolute": "עִ֣יר",
"absolute_consonants": "עיר"
},
{
"position": 12,
"construct": "שְׁנַ֣ת",
"construct_consonants": "שנת",
"absolute": "מִמְכָּר֑וֹ",
"absolute_consonants": "ממכרו"
}
]
}
+9
View File
@@ -237,5 +237,14 @@
"prefix": "ב",
"shoresh": "יבל"
}
],
"ketiv_qere": [
{
"word_position": 14,
"ketiv": "לא",
"ketiv_consonants": "לא",
"qere": "ל֣וֹ",
"qere_consonants": "לו"
}
]
}
+16
View File
@@ -174,5 +174,21 @@
],
"shoresh": "יצא"
}
],
"smikhut": [
{
"position": 1,
"construct": "וּבָתֵּ֣י",
"construct_consonants": "ובתי",
"absolute": "הַחֲצֵרִ֗ים",
"absolute_consonants": "החצרים"
},
{
"position": 9,
"construct": "שְׂדֵ֥ה",
"construct_consonants": "שדה",
"absolute": "הָאָ֖רֶץ",
"absolute_consonants": "הארץ"
}
]
}
+30
View File
@@ -106,5 +106,35 @@
"prefix": "ל",
"shoresh": "לו"
}
],
"smikhut": [
{
"position": 1,
"construct": "וְעָרֵי֙",
"construct_consonants": "וערי",
"absolute": "הַלְוִיִּ֔ם",
"absolute_consonants": "הלוים"
},
{
"position": 3,
"construct": "בָּתֵּ֖י",
"construct_consonants": "בתי",
"absolute": "עָרֵ֣י",
"absolute_consonants": "ערי"
},
{
"position": 4,
"construct": "עָרֵ֣י",
"construct_consonants": "ערי",
"absolute": "אֲחֻזָּתָ֑ם",
"absolute_consonants": "אחזתם"
},
{
"position": 6,
"construct": "גְּאֻלַּ֥ת",
"construct_consonants": "גאלת",
"absolute": "עוֹלָ֖ם",
"absolute_consonants": "עולם"
}
]
}
+44
View File
@@ -211,5 +211,49 @@
],
"shoresh": "ישר"
}
],
"smikhut": [
{
"position": 6,
"construct": "מִמְכַּר",
"construct_consonants": "ממכר",
"absolute": "בַּ֛יִת",
"absolute_consonants": "בית"
},
{
"position": 8,
"construct": "וְעִ֥יר",
"construct_consonants": "ועיר",
"absolute": "אֲחֻזָּת֖וֹ",
"absolute_consonants": "אחזתו"
},
{
"position": 12,
"construct": "בָתֵּ֞י",
"construct_consonants": "בתי",
"absolute": "עָרֵ֣י",
"absolute_consonants": "ערי"
},
{
"position": 13,
"construct": "עָרֵ֣י",
"construct_consonants": "ערי",
"absolute": "הַלְוִיִּ֗ם",
"absolute_consonants": "הלוים"
},
{
"position": 17,
"construct": "בְּת֖וֹךְ",
"construct_consonants": "בתוך",
"absolute": "בְּנֵ֥י",
"absolute_consonants": "בני"
},
{
"position": 18,
"construct": "בְּנֵ֥י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵֽל",
"absolute_consonants": "ישראל"
}
]
}
+23
View File
@@ -112,5 +112,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 1,
"construct": "וּֽשְׂדֵ֛ה",
"construct_consonants": "ושדה",
"absolute": "מִגְרַ֥שׁ",
"absolute_consonants": "מגרש"
},
{
"position": 2,
"construct": "מִגְרַ֥שׁ",
"construct_consonants": "מגרש",
"absolute": "עָרֵיהֶ֖ם",
"absolute_consonants": "עריהם"
},
{
"position": 7,
"construct": "אֲחֻזַּ֥ת",
"construct_consonants": "אחזת",
"absolute": "עוֹלָ֛ם",
"absolute_consonants": "עולם"
}
]
}
+16
View File
@@ -175,5 +175,21 @@
"prefix": "ל",
"shoresh": "אלה"
}
],
"smikhut": [
{
"position": 7,
"construct": "מֵאֶ֣רֶץ",
"construct_consonants": "מארץ",
"absolute": "מִצְרָ֑יִם",
"absolute_consonants": "מצרים"
},
{
"position": 12,
"construct": "אֶ֣רֶץ",
"construct_consonants": "ארץ",
"absolute": "כְּנַ֔עַן",
"absolute_consonants": "כנען"
}
]
}
+9
View File
@@ -118,5 +118,14 @@
],
"shoresh": "עבד"
}
],
"smikhut": [
{
"position": 10,
"construct": "עֲבֹ֥דַת",
"construct_consonants": "עבדת",
"absolute": "עָֽבֶד",
"absolute_consonants": "עבד"
}
]
}
+9
View File
@@ -161,5 +161,14 @@
],
"shoresh": "זמר"
}
],
"smikhut": [
{
"position": 3,
"construct": "שַׁבַּ֤ת",
"construct_consonants": "שבת",
"absolute": "שַׁבָּתוֹן֙",
"absolute_consonants": "שבתון"
}
]
}
+9
View File
@@ -102,5 +102,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 6,
"construct": "שְׁנַ֥ת",
"construct_consonants": "שנת",
"absolute": "הַיֹּבֵ֖ל",
"absolute_consonants": "היבל"
}
]
}
+9
View File
@@ -131,5 +131,14 @@
],
"shoresh": "שוב"
}
],
"smikhut": [
{
"position": 10,
"construct": "אֲחֻזַּ֥ת",
"construct_consonants": "אחזת",
"absolute": "אֲבֹתָ֖יו",
"absolute_consonants": "אבתיו"
}
]
}
+16
View File
@@ -130,5 +130,21 @@
],
"shoresh": "עבד"
}
],
"smikhut": [
{
"position": 7,
"construct": "מֵאֶ֣רֶץ",
"construct_consonants": "מארץ",
"absolute": "מִצְרָ֑יִם",
"absolute_consonants": "מצרים"
},
{
"position": 11,
"construct": "מִמְכֶּ֥רֶת",
"construct_consonants": "ממכרת",
"absolute": "עָֽבֶד",
"absolute_consonants": "עבד"
}
]
}
+9
View File
@@ -181,5 +181,14 @@
"prefix": "ל",
"shoresh": "אחז"
}
],
"smikhut": [
{
"position": 2,
"construct": "מִבְּנֵ֨י",
"construct_consonants": "מבני",
"absolute": "הַתּוֹשָׁבִ֜ים",
"absolute_consonants": "התושבים"
}
]
}
+9
View File
@@ -201,5 +201,14 @@
"prefix": "ב",
"shoresh": "פרך"
}
],
"smikhut": [
{
"position": 11,
"construct": "בְּנֵֽי",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵל֙",
"absolute_consonants": "ישראל"
}
]
}
+23
View File
@@ -192,5 +192,28 @@
],
"shoresh": "גר"
}
],
"smikhut": [
{
"position": 3,
"construct": "יַ֣ד",
"construct_consonants": "יד",
"absolute": "גֵּ֤ר",
"absolute_consonants": "גר"
},
{
"position": 15,
"construct": "לְעֵ֖קֶר",
"construct_consonants": "לעקר",
"absolute": "מִשְׁפַּ֥חַת",
"absolute_consonants": "משפחת"
},
{
"position": 16,
"construct": "מִשְׁפַּ֥חַת",
"construct_consonants": "משפחת",
"absolute": "גֵּֽר",
"absolute_consonants": "גר"
}
]
}
+16
View File
@@ -158,5 +158,21 @@
"prefix": "ו",
"shoresh": "גאל"
}
],
"smikhut": [
{
"position": 4,
"construct": "בֶן",
"construct_consonants": "בן",
"absolute": "דֹּדוֹ֙",
"absolute_consonants": "דדו"
},
{
"position": 8,
"construct": "מִשְּׁאֵ֧ר",
"construct_consonants": "משאר",
"absolute": "בְּשָׂר֛וֹ",
"absolute_consonants": "בשרו"
}
]
}
+23
View File
@@ -154,5 +154,28 @@
"prefix": "ל",
"shoresh": "ארץ"
}
],
"smikhut": [
{
"position": 2,
"construct": "סְפִ֤יחַ",
"construct_consonants": "ספיח",
"absolute": "קְצִֽירְךָ֙",
"absolute_consonants": "קצירך"
},
{
"position": 7,
"construct": "עִנְּבֵ֥י",
"construct_consonants": "ענבי",
"absolute": "נְזִירֶ֖ךָ",
"absolute_consonants": "נזירך"
},
{
"position": 11,
"construct": "שְׁנַ֥ת",
"construct_consonants": "שנת",
"absolute": "שַׁבָּת֖וֹן",
"absolute_consonants": "שבתון"
}
]
}
+37
View File
@@ -200,5 +200,42 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 4,
"construct": "מִשְּׁנַת֙",
"construct_consonants": "משנת",
"absolute": "הִמָּ֣כְרוֹ",
"absolute_consonants": "המכרו"
},
{
"position": 8,
"construct": "שְׁנַ֣ת",
"construct_consonants": "שנת",
"absolute": "הַיֹּבֵ֑ל",
"absolute_consonants": "היבל"
},
{
"position": 11,
"construct": "כֶּ֤סֶף",
"construct_consonants": "כסף",
"absolute": "מִמְכָּרוֹ֙",
"absolute_consonants": "ממכרו"
},
{
"position": 13,
"construct": "בְּמִסְפַּ֣ר",
"construct_consonants": "במספר",
"absolute": "שָׁנִ֔ים",
"absolute_consonants": "שנים"
},
{
"position": 15,
"construct": "כִּימֵ֥י",
"construct_consonants": "כימי",
"absolute": "שָׂכִ֖יר",
"absolute_consonants": "שכיר"
}
]
}
+9
View File
@@ -101,5 +101,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 8,
"construct": "מִכֶּ֖סֶף",
"construct_consonants": "מכסף",
"absolute": "מִקְנָתֽוֹ",
"absolute_consonants": "מקנתו"
}
]
}
+16
View File
@@ -150,5 +150,21 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 6,
"construct": "שְׁנַ֥ת",
"construct_consonants": "שנת",
"absolute": "הַיֹּבֵ֖ל",
"absolute_consonants": "היבל"
},
{
"position": 10,
"construct": "כְּפִ֣י",
"construct_consonants": "כפי",
"absolute": "שָׁנָ֔יו",
"absolute_consonants": "שניו"
}
]
}
+9
View File
@@ -105,5 +105,14 @@
],
"prefix": "ל"
}
],
"smikhut": [
{
"position": 1,
"construct": "כִּשְׂכִ֥יר",
"construct_consonants": "כשכיר",
"absolute": "שָׁנָ֛ה",
"absolute_consonants": "שנה"
}
]
}
+9
View File
@@ -113,5 +113,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 6,
"construct": "בִּשְׁנַ֣ת",
"construct_consonants": "בשנת",
"absolute": "הַיֹּבֵ֔ל",
"absolute_consonants": "היבל"
}
]
}
+16
View File
@@ -161,5 +161,21 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 3,
"construct": "בְנֵֽי",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵל֙",
"absolute_consonants": "ישראל"
},
{
"position": 11,
"construct": "מֵאֶ֣רֶץ",
"construct_consonants": "מארץ",
"absolute": "מִצְרָ֑יִם",
"absolute_consonants": "מצרים"
}
]
}
+9
View File
@@ -140,5 +140,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 2,
"construct": "שַׁבַּ֨ת",
"construct_consonants": "שבת",
"absolute": "הָאָ֤רֶץ",
"absolute_consonants": "הארץ"
}
]
}
+9
View File
@@ -93,5 +93,14 @@
"prefix": "ל",
"shoresh": "אכל"
}
],
"smikhut": [
{
"position": 6,
"construct": "כָל",
"construct_consonants": "כל",
"absolute": "תְּבוּאָתָ֖הּ",
"absolute_consonants": "תבואתה"
}
]
}
+23
View File
@@ -205,5 +205,28 @@
],
"shoresh": "שנה"
}
],
"smikhut": [
{
"position": 4,
"construct": "שַׁבְּתֹ֣ת",
"construct_consonants": "שבתת",
"absolute": "שָׁנִ֔ים",
"absolute_consonants": "שנים"
},
{
"position": 12,
"construct": "יְמֵי֙",
"construct_consonants": "ימי",
"absolute": "שֶׁ֚בַע",
"absolute_consonants": "שבע"
},
{
"position": 14,
"construct": "שַׁבְּתֹ֣ת",
"construct_consonants": "שבתת",
"absolute": "הַשָּׁנִ֔ים",
"absolute_consonants": "השנים"
}
]
}
+23
View File
@@ -154,5 +154,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 2,
"construct": "שׁוֹפַ֤ר",
"construct_consonants": "שופר",
"absolute": "תְּרוּעָה֙",
"absolute_consonants": "תרועה"
},
{
"position": 8,
"construct": "בְּיוֹם֙",
"construct_consonants": "ביום",
"absolute": "הַכִּפֻּרִ֔ים",
"absolute_consonants": "הכפרים"
},
{
"position": 12,
"construct": "בְּכָל",
"construct_consonants": "בכל",
"absolute": "אַרְצְכֶֽם",
"absolute_consonants": "ארצכם"
}
]
}