L1 data integrity: preserve the edut whole — Shema, middle letter, ketiv/qere, smikhut

Shemot 25:16 — v'natata el ha-aron et ha-edut asher eten elekha.
The edut placed in the aron must be complete. The Masoretic
tradition preserves every letter, every enlarged letter, every
ketiv/qere pair — because each carries meaning the scribe cannot
reconstruct.

The previous build_l1.py silently dropped:
- enlarged letters encoded as <seg type="x-large"> nested in <w>
  → the Shema was corrupt: שמ instead of שמע, אח instead of אחד
  → the middle letter of Torah was corrupt: גח instead of גחון
  → the Zelophchad nun was missing: משפט instead of משפטן
- ketiv/qere dual readings encoded as <note type="variant">/<rdg type="x-qere">
  → the spec committed to preserving both; the code returned []
- smikhut (construct state) pairs marked in morphology codes
  → the spec committed to extracting them; the code returned []

Fixes:
- collect_word_text() walks all nested <seg> children of <w> and records
  large_letters / small_letters metadata with character offsets.
- parse_verse() recognizes <w type="x-ketiv"> followed by <note>, extracts
  both the ketiv and the qere readings, stores a ketiv_qere list on the verse.
- _extract_smikhut() scans words for construct-state morph codes
  (ending in "c" on N/A primary segments, excluding pronominal suffixes)
  and records construct/absolute pairs.
- L1 SHA-256 hash updated everywhere: integrity.sha256, l1_reader.ex,
  shoresh-engine's Luchot.Firmware, its test, KOHEN.md.

Torah-wide results after fix:
- 4 enlarged letters preserved (was 0): Shema ayin, Shema dalet,
  gachon vav (middle of Torah), Zelophchad nun
- 67 ketiv/qere pairs captured (was 0)
- 9,203 smikhut pairs extracted (was 0)

Elixir updates:
- l1_reader.ex: normalize_verse now surfaces ketiv_qere, smikhut,
  and per-word large_letters/small_letters. New type specs. New
  extract_large_letters/1 helper. extract_ketiv_qere and
  extract_smikhut now return real data instead of [].
- step31_shema_test.exs: replaced placeholder asserts with real
  integrity checks. Four new tests verify the Shema word is complete
  (שמע not שמ), echad is complete (אחד not אח), the enlarged
  letters are marked, the middle letter of Torah is preserved,
  and ketiv/qere pairs from Bereishit 8:17 are both captured.

Build script:
- build_l1.py no longer rmtrees the entire l1/ directory (preserves
  l1/README.md). Only regenerates torah/ + build_report.json +
  integrity.sha256.
- Accepts SHORASHIM_SOURCE_DATA env var or CLI arg for morphhb path.

Test counts:
- Elixir: 260 (was 254), 0 failures
- Haskell: 33, 0 failures
- Prolog: 195, 0 failures

The system that can hear שמע correctly has passed its own shema test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Opus 4.6 committed 2026-04-21 13:11:52 -04:00
1 parent 04713cc884
commit d4d0a6fcb6
4137 files changed
+73656 -85

No files matched your search

+16
View File
@@ -72,5 +72,21 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 2,
"construct": "תּוֹרַ֖ת",
"construct_consonants": "תורת",
"absolute": "הָאָשָׁ֑ם",
"absolute_consonants": "האשם"
},
{
"position": 4,
"construct": "קֹ֥דֶשׁ",
"construct_consonants": "קדש",
"absolute": "קָֽדָשִׁ֖ים",
"absolute_consonants": "קדשים"
}
]
}
+23
View File
@@ -126,5 +126,28 @@
],
"prefix": "כ"
}
],
"smikhut": [
{
"position": 1,
"construct": "וְכָל",
"construct_consonants": "וכל",
"absolute": "מִנְחָ֥ה",
"absolute_consonants": "מנחה"
},
{
"position": 6,
"construct": "לְכָל",
"construct_consonants": "לכל",
"absolute": "בְּנֵ֧י",
"absolute_consonants": "בני"
},
{
"position": 7,
"construct": "בְּנֵ֧י",
"construct_consonants": "בני",
"absolute": "אַהֲרֹ֛ן",
"absolute_consonants": "אהרן"
}
]
}
+16
View File
@@ -83,5 +83,21 @@
"prefix": "ל",
"shoresh": "יהו"
}
],
"smikhut": [
{
"position": 2,
"construct": "תּוֹרַ֖ת",
"construct_consonants": "תורת",
"absolute": "זֶ֣בַח",
"absolute_consonants": "זבח"
},
{
"position": 3,
"construct": "זֶ֣בַח",
"construct_consonants": "זבח",
"absolute": "הַשְּׁלָמִ֑ים",
"absolute_consonants": "השלמים"
}
]
}
+16
View File
@@ -235,5 +235,21 @@
"prefix": "ב",
"shoresh": "שמן"
}
],
"smikhut": [
{
"position": 7,
"construct": "זֶ֣בַח",
"construct_consonants": "זבח",
"absolute": "הַתּוֹדָ֗ה",
"absolute_consonants": "התודה"
},
{
"position": 13,
"construct": "וּרְקִיקֵ֥י",
"construct_consonants": "ורקיקי",
"absolute": "מַצּ֖וֹת",
"absolute_consonants": "מצות"
}
]
}
+23
View File
@@ -107,5 +107,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 2,
"construct": "חַלֹּת֙",
"construct_consonants": "חלת",
"absolute": "לֶ֣חֶם",
"absolute_consonants": "לחם"
},
{
"position": 3,
"construct": "לֶ֣חֶם",
"construct_consonants": "לחם",
"absolute": "חָמֵ֔ץ",
"absolute_consonants": "חמץ"
},
{
"position": 9,
"construct": "תּוֹדַ֥ת",
"construct_consonants": "תודת",
"absolute": "שְׁלָמָֽיו",
"absolute_consonants": "שלמיו"
}
]
}
+16
View File
@@ -158,5 +158,21 @@
],
"shoresh": "היה"
}
],
"smikhut": [
{
"position": 4,
"construct": "מִכָּל",
"construct_consonants": "מכל",
"absolute": "קָרְבָּ֔ן",
"absolute_consonants": "קרבן"
},
{
"position": 11,
"construct": "דַּ֥ם",
"construct_consonants": "דם",
"absolute": "הַשְּׁלָמִ֖ים",
"absolute_consonants": "השלמים"
}
]
}
+30
View File
@@ -132,5 +132,35 @@
],
"shoresh": "בקר"
}
],
"smikhut": [
{
"position": 1,
"construct": "וּבְשַׂ֗ר",
"construct_consonants": "ובשר",
"absolute": "זֶ֚בַח",
"absolute_consonants": "זבח"
},
{
"position": 2,
"construct": "זֶ֚בַח",
"construct_consonants": "זבח",
"absolute": "תּוֹדַ֣ת",
"absolute_consonants": "תודת"
},
{
"position": 3,
"construct": "תּוֹדַ֣ת",
"construct_consonants": "תודת",
"absolute": "שְׁלָמָ֔יו",
"absolute_consonants": "שלמיו"
},
{
"position": 5,
"construct": "בְּי֥וֹם",
"construct_consonants": "ביום",
"absolute": "קָרְבָּנ֖וֹ",
"absolute_consonants": "קרבנו"
}
]
}
+16
View File
@@ -164,5 +164,21 @@
],
"shoresh": "אכל"
}
],
"smikhut": [
{
"position": 5,
"construct": "זֶ֚בַח",
"construct_consonants": "זבח",
"absolute": "קָרְבָּנ֔וֹ",
"absolute_consonants": "קרבנו"
},
{
"position": 7,
"construct": "בְּי֛וֹם",
"construct_consonants": "ביום",
"absolute": "הַקְרִיב֥וֹ",
"absolute_consonants": "הקריבו"
}
]
}
+9
View File
@@ -88,5 +88,14 @@
],
"shoresh": "שרף"
}
],
"smikhut": [
{
"position": 2,
"construct": "מִבְּשַׂ֣ר",
"construct_consonants": "מבשר",
"absolute": "הַזָּ֑בַח",
"absolute_consonants": "הזבח"
}
]
}
+16
View File
@@ -244,5 +244,21 @@
],
"shoresh": "תשא"
}
],
"smikhut": [
{
"position": 4,
"construct": "מִבְּשַׂר",
"construct_consonants": "מבשר",
"absolute": "זֶ֨בַח",
"absolute_consonants": "זבח"
},
{
"position": 5,
"construct": "זֶ֨בַח",
"construct_consonants": "זבח",
"absolute": "שְׁלָמָ֜יו",
"absolute_consonants": "שלמיו"
}
]
}
+16
View File
@@ -157,5 +157,21 @@
],
"shoresh": "בשר"
}
],
"smikhut": [
{
"position": 4,
"construct": "בְּכָל",
"construct_consonants": "בכל",
"absolute": "טָמֵא֙",
"absolute_consonants": "טמא"
},
{
"position": 11,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "טָה֖וֹר",
"absolute_consonants": "טהור"
}
]
}
+9
View File
@@ -151,5 +151,14 @@
],
"shoresh": "סבי"
}
],
"smikhut": [
{
"position": 1,
"construct": "בִּמְק֗וֹם",
"construct_consonants": "במקום",
"absolute": "אֲשֶׁ֤ר",
"absolute_consonants": "אשר"
}
]
}
+9
View File
@@ -160,5 +160,14 @@
],
"prefix": "מ"
}
],
"smikhut": [
{
"position": 5,
"construct": "מִזֶּ֤בַח",
"construct_consonants": "מזבח",
"absolute": "הַשְּׁלָמִים֙",
"absolute_consonants": "השלמים"
}
]
}
+37
View File
@@ -270,5 +270,42 @@
],
"prefix": "מ"
}
],
"smikhut": [
{
"position": 4,
"construct": "בְּכָל",
"construct_consonants": "בכל",
"absolute": "טָמֵ֗א",
"absolute_consonants": "טמא"
},
{
"position": 6,
"construct": "בְּטֻמְאַ֤ת",
"construct_consonants": "בטמאת",
"absolute": "אָדָם֙",
"absolute_consonants": "אדם"
},
{
"position": 12,
"construct": "בְּכָל",
"construct_consonants": "בכל",
"absolute": "שֶׁ֣קֶץ",
"absolute_consonants": "שקץ"
},
{
"position": 16,
"construct": "מִבְּשַׂר",
"construct_consonants": "מבשר",
"absolute": "זֶ֥בַח",
"absolute_consonants": "זבח"
},
{
"position": 17,
"construct": "זֶ֥בַח",
"construct_consonants": "זבח",
"absolute": "הַשְּׁלָמִ֖ים",
"absolute_consonants": "השלמים"
}
]
}
+23
View File
@@ -134,5 +134,28 @@
],
"shoresh": "אכל"
}
],
"smikhut": [
{
"position": 3,
"construct": "בְּנֵ֥י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵ֖ל",
"absolute_consonants": "ישראל"
},
{
"position": 6,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "חֵ֜לֶב",
"absolute_consonants": "חלב"
},
{
"position": 7,
"construct": "חֵ֜לֶב",
"construct_consonants": "חלב",
"absolute": "שׁ֥וֹר",
"absolute_consonants": "שור"
}
]
}
+23
View File
@@ -115,5 +115,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 1,
"construct": "וְחֵ֤לֶב",
"construct_consonants": "וחלב",
"absolute": "נְבֵלָה֙",
"absolute_consonants": "נבלה"
},
{
"position": 3,
"construct": "וְחֵ֣לֶב",
"construct_consonants": "וחלב",
"absolute": "טְרֵפָ֔ה",
"absolute_consonants": "טרפה"
},
{
"position": 6,
"construct": "לְכָל",
"construct_consonants": "לכל",
"absolute": "מְלָאכָ֑ה",
"absolute_consonants": "מלאכה"
}
]
}
+9
View File
@@ -169,5 +169,14 @@
],
"prefix": "מ"
}
],
"smikhut": [
{
"position": 2,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "אֹכֵ֣ל",
"absolute_consonants": "אכל"
}
]
}
+16
View File
@@ -93,5 +93,21 @@
"prefix": "ול",
"shoresh": "בהמ"
}
],
"smikhut": [
{
"position": 1,
"construct": "וְכָל",
"construct_consonants": "וכל",
"absolute": "דָּם֙",
"absolute_consonants": "דם"
},
{
"position": 5,
"construct": "בְּכֹ֖ל",
"construct_consonants": "בכל",
"absolute": "מוֹשְׁבֹתֵיכֶ֑ם",
"absolute_consonants": "מושבתיכם"
}
]
}
+16
View File
@@ -111,5 +111,21 @@
],
"prefix": "מ"
}
],
"smikhut": [
{
"position": 1,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "נֶ֖פֶשׁ",
"absolute_consonants": "נפש"
},
{
"position": 5,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "דָּ֑ם",
"absolute_consonants": "דם"
}
]
}
+23
View File
@@ -174,5 +174,28 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 3,
"construct": "בְּנֵ֥י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵ֖ל",
"absolute_consonants": "ישראל"
},
{
"position": 8,
"construct": "זֶ֤בַח",
"construct_consonants": "זבח",
"absolute": "שְׁלָמָיו֙",
"absolute_consonants": "שלמיו"
},
{
"position": 15,
"construct": "מִזֶּ֥בַח",
"construct_consonants": "מזבח",
"absolute": "שְׁלָמָֽיו",
"absolute_consonants": "שלמיו"
}
]
}
+9
View File
@@ -133,5 +133,14 @@
"prefix": "ה",
"shoresh": "קרב"
}
],
"smikhut": [
{
"position": 2,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "חֶלְבּ֖וֹ",
"absolute_consonants": "חלבו"
}
]
}
+16
View File
@@ -187,5 +187,21 @@
],
"shoresh": "יהו"
}
],
"smikhut": [
{
"position": 4,
"construct": "אִשֵּׁ֣י",
"construct_consonants": "אשי",
"absolute": "יְהוָ֑ה",
"absolute_consonants": "יהוה"
},
{
"position": 16,
"construct": "לִפְנֵ֥י",
"construct_consonants": "לפני",
"absolute": "יְהוָֽה",
"absolute_consonants": "יהוה"
}
]
}
+16
View File
@@ -95,5 +95,21 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 2,
"construct": "שׁ֣וֹק",
"construct_consonants": "שוק",
"absolute": "הַיָּמִ֔ין",
"absolute_consonants": "הימין"
},
{
"position": 7,
"construct": "מִזִּבְחֵ֖י",
"construct_consonants": "מזבחי",
"absolute": "שַׁלְמֵיכֶֽם",
"absolute_consonants": "שלמיכם"
}
]
}
+23
View File
@@ -148,5 +148,28 @@
"prefix": "ל",
"shoresh": "מנה"
}
],
"smikhut": [
{
"position": 3,
"construct": "דַּ֧ם",
"construct_consonants": "דם",
"absolute": "הַשְּׁלָמִ֛ים",
"absolute_consonants": "השלמים"
},
{
"position": 7,
"construct": "מִבְּנֵ֣י",
"construct_consonants": "מבני",
"absolute": "אַהֲרֹ֑ן",
"absolute_consonants": "אהרן"
},
{
"position": 11,
"construct": "שׁ֥וֹק",
"construct_consonants": "שוק",
"absolute": "הַיָּמִ֖ין",
"absolute_consonants": "הימין"
}
]
}
+44
View File
@@ -258,5 +258,49 @@
],
"shoresh": "ישר"
}
],
"smikhut": [
{
"position": 3,
"construct": "חֲזֵ֨ה",
"construct_consonants": "חזה",
"absolute": "הַתְּנוּפָ֜ה",
"absolute_consonants": "התנופה"
},
{
"position": 6,
"construct": "שׁ֣וֹק",
"construct_consonants": "שוק",
"absolute": "הַתְּרוּמָ֗ה",
"absolute_consonants": "התרומה"
},
{
"position": 10,
"construct": "בְּנֵֽי",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵ֔ל",
"absolute_consonants": "ישראל"
},
{
"position": 12,
"construct": "מִזִּבְחֵ֖י",
"construct_consonants": "מזבחי",
"absolute": "שַׁלְמֵיהֶ֑ם",
"absolute_consonants": "שלמיהם"
},
{
"position": 19,
"construct": "לְחָק",
"construct_consonants": "לחק",
"absolute": "עוֹלָ֔ם",
"absolute_consonants": "עולם"
},
{
"position": 22,
"construct": "בְּנֵ֥י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵֽל",
"absolute_consonants": "ישראל"
}
]
}
+30
View File
@@ -139,5 +139,35 @@
"prefix": "ל",
"shoresh": "יהו"
}
],
"smikhut": [
{
"position": 2,
"construct": "מִשְׁחַ֤ת",
"construct_consonants": "משחת",
"absolute": "אַהֲרֹן֙",
"absolute_consonants": "אהרן"
},
{
"position": 4,
"construct": "וּמִשְׁחַ֣ת",
"construct_consonants": "ומשחת",
"absolute": "בָּנָ֔יו",
"absolute_consonants": "בניו"
},
{
"position": 6,
"construct": "מֵאִשֵּׁ֖י",
"construct_consonants": "מאשי",
"absolute": "יְהוָ֑ה",
"absolute_consonants": "יהוה"
},
{
"position": 8,
"construct": "בְּיוֹם֙",
"construct_consonants": "ביום",
"absolute": "הִקְרִ֣יב",
"absolute_consonants": "הקריב"
}
]
}
+23
View File
@@ -157,5 +157,28 @@
],
"prefix": "ל"
}
],
"smikhut": [
{
"position": 6,
"construct": "בְּיוֹם֙",
"construct_consonants": "ביום",
"absolute": "מָשְׁח֣וֹ",
"absolute_consonants": "משחו"
},
{
"position": 10,
"construct": "בְּנֵ֣י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵ֑ל",
"absolute_consonants": "ישראל"
},
{
"position": 12,
"construct": "חֻקַּ֥ת",
"construct_consonants": "חקת",
"absolute": "עוֹלָ֖ם",
"absolute_consonants": "עולם"
}
]
}
+9
View File
@@ -114,5 +114,14 @@
"prefix": "ה",
"shoresh": "שלמ"
}
],
"smikhut": [
{
"position": 8,
"construct": "וּלְזֶ֖בַח",
"construct_consonants": "ולזבח",
"absolute": "הַשְּׁלָמִֽים",
"absolute_consonants": "השלמים"
}
]
}
+30
View File
@@ -196,5 +196,35 @@
],
"shoresh": "סינ"
}
],
"smikhut": [
{
"position": 6,
"construct": "בְּהַ֣ר",
"construct_consonants": "בהר",
"absolute": "סִינָ֑י",
"absolute_consonants": "סיני"
},
{
"position": 8,
"construct": "בְּי֨וֹם",
"construct_consonants": "ביום",
"absolute": "צַוֺּת֜וֹ",
"absolute_consonants": "צותו"
},
{
"position": 11,
"construct": "בְּנֵ֣י",
"construct_consonants": "בני",
"absolute": "יִשְׂרָאֵ֗ל",
"absolute_consonants": "ישראל"
},
{
"position": 17,
"construct": "בְּמִדְבַּ֥ר",
"construct_consonants": "במדבר",
"absolute": "סִינָֽי",
"absolute_consonants": "סיני"
}
]
}
+9
View File
@@ -183,5 +183,14 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 2,
"construct": "שְׁתֵּ֣י",
"construct_consonants": "שתי",
"absolute": "הַכְּלָיֹ֔ת",
"absolute_consonants": "הכלית"
}
]
}
+16
View File
@@ -114,5 +114,21 @@
"meteg_siluq"
]
}
],
"smikhut": [
{
"position": 1,
"construct": "כָּל",
"construct_consonants": "כל",
"absolute": "זָכָ֥ר",
"absolute_consonants": "זכר"
},
{
"position": 8,
"construct": "קֹ֥דֶשׁ",
"construct_consonants": "קדש",
"absolute": "קָֽדָשִׁ֖ים",
"absolute_consonants": "קדשים"
}
]
}
+16
View File
@@ -138,5 +138,21 @@
],
"shoresh": "היה"
}
],
"smikhut": [
{
"position": 4,
"construct": "עֹ֣לַת",
"construct_consonants": "עלת",
"absolute": "אִ֑ישׁ",
"absolute_consonants": "איש"
},
{
"position": 6,
"construct": "ע֤וֹר",
"construct_consonants": "עור",
"absolute": "הָֽעֹלָה֙",
"absolute_consonants": "העלה"
}
]
}
+16
View File
@@ -171,5 +171,21 @@
],
"shoresh": "היה"
}
],
"smikhut": [
{
"position": 1,
"construct": "וְכָל",
"construct_consonants": "וכל",
"absolute": "מִנְחָ֗ה",
"absolute_consonants": "מנחה"
},
{
"position": 6,
"construct": "וְכָל",
"construct_consonants": "וכל",
"absolute": "נַעֲשָׂ֥ה",
"absolute_consonants": "נעשה"
}
]
}