L1 data integrity: preserve the edut whole — Shema, middle letter, ketiv/qere, smikhut

Shemot 25:16 — v'natata el ha-aron et ha-edut asher eten elekha.
The edut placed in the aron must be complete. The Masoretic
tradition preserves every letter, every enlarged letter, every
ketiv/qere pair — because each carries meaning the scribe cannot
reconstruct.

The previous build_l1.py silently dropped:
- enlarged letters encoded as <seg type="x-large"> nested in <w>
  → the Shema was corrupt: שמ instead of שמע, אח instead of אחד
  → the middle letter of Torah was corrupt: גח instead of גחון
  → the Zelophchad nun was missing: משפט instead of משפטן
- ketiv/qere dual readings encoded as <note type="variant">/<rdg type="x-qere">
  → the spec committed to preserving both; the code returned []
- smikhut (construct state) pairs marked in morphology codes
  → the spec committed to extracting them; the code returned []

Fixes:
- collect_word_text() walks all nested <seg> children of <w> and records
  large_letters / small_letters metadata with character offsets.
- parse_verse() recognizes <w type="x-ketiv"> followed by <note>, extracts
  both the ketiv and the qere readings, stores a ketiv_qere list on the verse.
- _extract_smikhut() scans words for construct-state morph codes
  (ending in "c" on N/A primary segments, excluding pronominal suffixes)
  and records construct/absolute pairs.
- L1 SHA-256 hash updated everywhere: integrity.sha256, l1_reader.ex,
  shoresh-engine's Luchot.Firmware, its test, KOHEN.md.

Torah-wide results after fix:
- 4 enlarged letters preserved (was 0): Shema ayin, Shema dalet,
  gachon vav (middle of Torah), Zelophchad nun
- 67 ketiv/qere pairs captured (was 0)
- 9,203 smikhut pairs extracted (was 0)

Elixir updates:
- l1_reader.ex: normalize_verse now surfaces ketiv_qere, smikhut,
  and per-word large_letters/small_letters. New type specs. New
  extract_large_letters/1 helper. extract_ketiv_qere and
  extract_smikhut now return real data instead of [].
- step31_shema_test.exs: replaced placeholder asserts with real
  integrity checks. Four new tests verify the Shema word is complete
  (שמע not שמ), echad is complete (אחד not אח), the enlarged
  letters are marked, the middle letter of Torah is preserved,
  and ketiv/qere pairs from Bereishit 8:17 are both captured.

Build script:
- build_l1.py no longer rmtrees the entire l1/ directory (preserves
  l1/README.md). Only regenerates torah/ + build_report.json +
  integrity.sha256.
- Accepts SHORASHIM_SOURCE_DATA env var or CLI arg for morphhb path.

Test counts:
- Elixir: 260 (was 254), 0 failures
- Haskell: 33, 0 failures
- Prolog: 195, 0 failures

The system that can hear שמע correctly has passed its own shema test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

scripts/build_l1.py

@@ -300,6 +300,50 @@ def derive_shoresh_from_word(consonants: str, morph_info: dict) -> str | None:
     return hebrew_only if hebrew_only else None
 
 
+def collect_word_text(w_elem) -> tuple[str, list[dict], list[dict]]:
+    """
+    Gather the full text of a <w> element including nested <seg> children.
+
+    Returns (full_text, large_letters, small_letters) where each letter entry is
+    {"letter": str, "offset": int} — the offset is the character position within
+    the full_text string (pre prefix-split).
+
+    The Masoretic tradition marks certain letters enlarged (otiot gedolot) or
+    reduced (otiot zeirot). WLC encodes these as <seg type="x-large"> or
+    <seg type="x-small"> nested inside <w>. The previous parser read only the
+    direct text of <w> and silently dropped these letters — corrupting the
+    Shema (שמ instead of שמע), the middle letter of Torah (גח instead of גחון),
+    and other halakhically significant forms.
+    """
+    parts = []
+    large = []
+    small = []
+
+    if w_elem.text:
+        parts.append(w_elem.text)
+
+    for child in w_elem:
+        tag = child.tag.replace(f"{{{OSIS_NS}}}", "")
+        if tag == "seg":
+            seg_type = child.get("type", "")
+            seg_text = child.text or ""
+            offset = sum(len(p) for p in parts)
+            parts.append(seg_text)
+            # Record letters stripped to consonants only — otiot gedolot/zeirot
+            # are a property of specific letters, not of vowels/cantillation.
+            letters = strip_vowels_and_cantillation(seg_text)
+            if seg_type == "x-large" and letters:
+                for i, ch in enumerate(letters):
+                    large.append({"letter": ch, "offset": offset + i})
+            elif seg_type == "x-small" and letters:
+                for i, ch in enumerate(letters):
+                    small.append({"letter": ch, "offset": offset + i})
+        if child.tail:
+            parts.append(child.tail)
+
+    return "".join(parts), large, small
+
+
 def parse_verse(verse_elem, book_osis: str, book_hebrew: str) -> dict:
     """Parse a single verse element into the L1 JSON format."""
     osis_id = verse_elem.get("osisID")  # e.g., "Gen.1.1"
@@ -310,16 +354,41 @@ def parse_verse(verse_elem, book_osis: str, book_hebrew: str) -> dict:
     words = []
     full_text_parts = []
     word_position = 0
+    ketiv_qere_entries = []
 
-    for elem in verse_elem:
+    children = list(verse_elem)
+    i = 0
+    while i < len(children):
+        elem = children[i]
         tag = elem.tag.replace(f"{{{OSIS_NS}}}", "")
 
         if tag == "w":
             word_position += 1
             word_data = parse_word(elem, word_position)
             words.append(word_data)
-            # For text_hebrew, use the joined text (no / separators)
-            full_text_parts.append((elem.text or "").replace("/", ""))
+            full_text_parts.append(word_data["text"])
+
+            # Ketiv/qere: when a <w type="x-ketiv"> is followed by
+            # <note type="variant"><rdg type="x-qere"><w>...</w></rdg></note>,
+            # the ketiv is what's written (in the scroll) and the qere is what's
+            # read aloud. The Masoretic tradition preserves BOTH. Neither can be
+            # silently collapsed — that would be choosing one reading over the other.
+            if elem.get("type") == "x-ketiv" and i + 1 < len(children):
+                nxt = children[i + 1]
+                nxt_tag = nxt.tag.replace(f"{{{OSIS_NS}}}", "")
+                if nxt_tag == "note" and nxt.get("type") == "variant":
+                    qere_text = _extract_qere(nxt)
+                    if qere_text:
+                        ketiv_qere_entries.append({
+                            "word_position": word_position,
+                            "ketiv": word_data["text"],
+                            "ketiv_consonants": word_data["consonants"],
+                            "qere": qere_text,
+                            "qere_consonants": strip_vowels_and_cantillation(
+                                qere_text.replace("/", "")
+                            ),
+                        })
+                    i += 1  # Skip the note element
 
         elif tag == "seg":
             seg_type = elem.get("type", "")
@@ -329,11 +398,8 @@ def parse_verse(verse_elem, book_osis: str, book_hebrew: str) -> dict:
                 full_text_parts.append("־")
             elif seg_type == "x-paseq":
                 full_text_parts.append(" ׀ ")
-            # Also check for trailing <w> elements after seg (maqaf-joined words)
 
-        # Handle mixed content: seg followed by w within verse
-        if elem.tail and elem.tail.strip():
-            pass  # tail text handled by next element
+        i += 1
 
     # Reconstruct full verse text
     text_hebrew = " ".join(full_text_parts).strip()
@@ -341,9 +407,14 @@ def parse_verse(verse_elem, book_osis: str, book_hebrew: str) -> dict:
     text_hebrew = text_hebrew.replace(" ־ ", "־").replace(" ־", "־").replace("־ ", "־")
     text_hebrew = re.sub(r"\s+", " ", text_hebrew).strip()
 
+    # Extract smikhut pairs — construct state + next noun.
+    # The WLC morph code ending in "c" marks construct state (smikhut).
+    # Example: "av hamon" (father of multitude) — av is construct, hamon absolute.
+    smikhut = _extract_smikhut(words)
+
     ref = f"{book_hebrew.capitalize()} {chapter}:{verse_num}"
 
-    return {
+    verse_out = {
         "ref": ref,
         "book": book_hebrew,
         "chapter": chapter,
@@ -351,11 +422,82 @@ def parse_verse(verse_elem, book_osis: str, book_hebrew: str) -> dict:
         "text_hebrew": text_hebrew,
         "words": words,
     }
+    if ketiv_qere_entries:
+        verse_out["ketiv_qere"] = ketiv_qere_entries
+    if smikhut:
+        verse_out["smikhut"] = smikhut
+    return verse_out
+
+
+def _extract_qere(note_elem) -> str:
+    """Extract the qere reading from a <note type="variant"> element."""
+    for rdg in note_elem:
+        rdg_tag = rdg.tag.replace(f"{{{OSIS_NS}}}", "")
+        if rdg_tag == "rdg" and rdg.get("type") == "x-qere":
+            # <rdg> contains one or more <w> children
+            qere_parts = []
+            for w in rdg:
+                w_tag = w.tag.replace(f"{{{OSIS_NS}}}", "")
+                if w_tag == "w":
+                    qere_text, _, _ = collect_word_text(w)
+                    qere_parts.append(qere_text.replace("/", ""))
+            if qere_parts:
+                return " ".join(qere_parts)
+    return ""
+
+
+def _extract_smikhut(words: list[dict]) -> list[dict]:
+    """
+    Extract smikhut (construct-state) pairs. When a noun or adjective is in
+    construct state (morph code ending in "c"), it binds to the following
+    word as an absolute. Example: "av hamon" (אב המון) — father of multitude.
+
+    We skip word-internal smikhut (pronominal suffixes like "av-i" = my father),
+    which are marked by /Sp... segments. Those are real smikhut but single-word,
+    not two-word pairs.
+    """
+    pairs = []
+    for i, w in enumerate(words[:-1]):
+        morph = w.get("morph", "")
+        if not morph:
+            continue
+        # Morph format: H + primary-POS + optional /prefix-or-suffix segments.
+        # Strip leading H (language marker) and find the primary word segment
+        # (the first segment describing N=noun or A=adjective).
+        segments = [s.lstrip("H") for s in morph.split("/")]
+        primary = None
+        for s in segments:
+            if s and s[0] in ("N", "A"):
+                primary = s
+                break
+        if not primary:
+            continue
+        # Skip if the same segment list has a pronominal suffix (Sp...) —
+        # that's word-internal smikhut, not a two-word pair.
+        if any(s.startswith("Sp") for s in segments):
+            continue
+        # Construct state is the last character of the primary morph.
+        if not primary.endswith("c"):
+            continue
+        next_w = words[i + 1]
+        pairs.append({
+            "position": w["position"],
+            "construct": w["text"],
+            "construct_consonants": w["consonants"],
+            "absolute": next_w["text"],
+            "absolute_consonants": next_w["consonants"],
+        })
+    return pairs
 
 
 def parse_word(w_elem, position: int) -> dict:
-    """Parse a single <w> element into word data."""
-    raw_text = w_elem.text or ""
+    """Parse a single <w> element into word data.
+
+    Collects the full text including nested <seg type="x-large"> and
+    <seg type="x-small"> children so enlarged/small Masoretic letters are
+    preserved rather than silently dropped (as they were before).
+    """
+    raw_text, large_letters, small_letters = collect_word_text(w_elem)
     lemma_str = w_elem.get("lemma", "")
     morph_str = w_elem.get("morph", "")
     word_id = w_elem.get("id", "")
@@ -363,10 +505,6 @@ def parse_word(w_elem, position: int) -> dict:
     # Parse prefixes
     prefixes, core_lemma, core_morph = parse_prefixes(lemma_str, morph_str)
 
-    # The displayed text includes prefix separators as /
-    # Remove the / for the display text but preserve for analysis
-    display_text = raw_text  # Keep as-is with / markers
-
     # Get text without prefix separators for the full word
     joined_text = raw_text.replace("/", "")
 
@@ -405,6 +543,15 @@ def parse_word(w_elem, position: int) -> dict:
     if shoresh:
         word_data["shoresh"] = shoresh
 
+    # Otiot gedolot / zeirot — preserved as metadata on the word.
+    # These are halakhically significant (the enlarged ayin/dalet of Shema
+    # spell "ed" — witness; the enlarged vav of gachon marks the middle
+    # letter of Torah; the small aleph of vayikra marks Moshe's humility).
+    if large_letters:
+        word_data["large_letters"] = large_letters
+    if small_letters:
+        word_data["small_letters"] = small_letters
+
     return word_data
 
 
@@ -655,10 +802,16 @@ if __name__ == "__main__":
         print("Or pass a path as the first argument, or set SHORASHIM_SOURCE_DATA.")
         sys.exit(1)
 
-    # Clean output directory
-    if output_dir.exists():
-        import shutil
-        shutil.rmtree(output_dir)
+    # Clean only the generated artifacts — preserve hand-written files like README.md.
+    # The torah/ tree is regenerated. build_report.json and integrity.sha256 too.
+    import shutil
+    torah_dir = output_dir / "torah"
+    if torah_dir.exists():
+        shutil.rmtree(torah_dir)
+    for artifact in ("build_report.json", "integrity.sha256"):
+        p = output_dir / artifact
+        if p.exists():
+            p.unlink()
 
     os.makedirs(output_dir, exist_ok=True)