heaviness510/bikoret
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

L1: the shoresh is read from the lexicon, not guessed (tablet 34)

Browse Source 
L1's `shoresh` field was derived by stripping a prefix letter and a
suffix letter from the consonants and keeping the first three. וַיַּרְא
"and he saw" was engraved as ירא ("fear"); ויהי as יהי; שמים as שמ;
מזבח as מזב; הבאתי as בא. And parse_prefixes took a word's morphology
from the LAST morph segment, which for a suffixed word is the pronominal
suffix — so every suffixed noun and verb in Torah (לבבך, נפשך) was
classed as a suffix and written with no shoresh at all.

Every WLC word already carries its lemma, and the OSHB lexical index
files each lemma under its Brown-Driver-Briggs root (terumah §E2 named
BDB + Strong's as the root lexicon and marked it "needs consolidation").
scripts/build_l1.py now reads the root from that filing:

  main    the lemma is a root            23,750 words   וירא → ראה
  sub     filed under a root             31,066 words   מזבח → זבח
  single  stands alone in the lexicon     2,671 words   משה → משה
  none    no etymology recorded → absent  3,002 words   יוסף, מצרים
  particles, pronouns → absent, as before 19,493 words  את, אשר, כי

57,487 words carry a shoresh (was 51,836); 1,378 unique roots (was
2,608 fragments). Roots are written as the lexicon writes them — two
letters (בן), three, or more (שמאל).

WHAT CHANGED IN THE EDUT: only `shoresh`. Verified file by file — with
that one field removed, all 5,853 verse files are byte-identical to the
previous L1. Text, consonants, lemma, morph, te'amim, prefix, ketiv/
qere, large and small letters: untouched. The old builder, run against
the same morphhb checkout, reproduces the old hash exactly; the new
builder is deterministic (two runs, one hash).

  L1 SHA-256  99bbd07a…4ea377  →  156b1d27…d8c9d5
  sources     morphhb 3d15126 · HebrewLexicon 21c9add
              (recorded in l1/build_report.json)

Moved together, because they pin the hash or the count:
vav_pipeline L1Reader @l1_expected_hash · shoresh-engine Luchot.Firmware
(hash, uniqueRootCount) + Spec · azazel_processor goral._L1_INTEGRITY_HASH
· KOHEN.md / KOHEN_PROMPT.md gate · l1/README.md (new "Roots" section) ·
README.md (second quarry) · AGENTS.md · root_meanings.pl comment.
The Doorpost manifest is not touched; its hash does not move.

Note for the kohen: the goral's seed inherits the L1 hash, so lots cast
after this commit are cast from a different seed than the azazel
records already in L3. Those records are history and stand as written.

All five suites on the new L1: Elixir 461/0 (seeds 0 42 191243) ·
Haskell 55/0 · Prolog 6 suites exit 0 (swipl 9.2.2) · korban 222 passed
1 skipped · azazel 125 passed. Four images built; full local stack
(real engine binary, PostgreSQL 16 with SSL, migrations) answers
Bereishit 1:4 with ר-א-ה, health healthy, 6 interpretations in L3,
0 write failures.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Fable 5.1 committed 2026-10-04 17:44:02 +00:00
1 parent 6d3411fcfa
commit 8e4f94c66f
5846 files changed
+45816 -39402
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/build_l1.py
+204 -97
View File
@@ -8,6 +8,12 @@ the read-only Torah filesystem for Shorashim.
This is a construction tool. It runs once. It produces JSON files.
The consumers (Haskell L0, Erlang Shoresh Engine) will read these files.
Two quarries, both outside the repo (see __main__):
morphhb — the text: WLC with te'amim, lemma and morphology.
HebrewLexicon — the roots: the OSHB lexical index, which arranges
every Strong's lemma under its Brown-Driver-Briggs
root (specs/shorashim_terumah.md §E2).
Ka'asher tzivah Hashem et Moshe.
"""
@@ -149,12 +155,20 @@ def parse_prefixes(lemma_str: str, morph_str: str) -> tuple[list[str], str, str]
if part in PREFIX_MAP:
prefixes.append(PREFIX_MAP[part])
# Parse morph — strip leading 'H' and split on '/'
# Parse morph — strip the language letter and split on '/'.
#
# The segments run [prefix…, WORD, pronominal-suffix?] and the lemma
# has one part per prefix plus the word. So the word's own morphology
# is the segment at the same index as the lemma's last part — NOT the
# last segment. Until tablet 34 this took the last segment, which for
# לְבָבְךָ (HNcmsc/Sp2ms) is the suffix "Sp2ms": the word was classed
# as a suffix, and every suffixed noun and verb in Torah — 28,146
# words — was written without a shoresh.
core_morph = morph_str
if morph_str.startswith("H"):
morph_inner = morph_str[1:]
morph_parts = morph_inner.split("/")
core_morph = morph_parts[-1] # Last part is the core morphology
if morph_str[:1] in ("H", "A"):
morph_parts = morph_str[1:].split("/")
word_index = len(lemma_parts) - 1
core_morph = morph_parts[word_index] if word_index < len(morph_parts) else morph_parts[-1]
return prefixes, core_lemma, core_morph
@@ -226,78 +240,123 @@ def strong_to_lemma_text(strong_num: str) -> str:
return strong_num.strip()
def derive_shoresh_from_word(consonants: str, morph_info: dict) -> str | None:
"""
Attempt to derive a triliteral root from consonantal text and morphology.
# === THE ROOT LEXICON ===
#
# Broken tablet 34 (2026-10-04). Until now `shoresh` was guessed from the
# consonants: strip a prefix letter, strip a suffix letter, keep the first
# three. וַיַּרְא "and he saw" was engraved as ירא ("fear"); וַיְהִי as
# יהי; שָׁמַיִם as שמ; מִזְבֵּחַ as מזב; הֵבֵאתִי as בא. 2,608 "unique
# roots", most of them fragments.
#
# Every word in the WLC already carries its lemma — a Strong's number —
# and the OSHB lexical index files each lemma under its Brown-Driver-
# Briggs root. The root is not derived here. It is read.
#
# <entry id="gqv"> מִזְבֵּחַ strong="4196" <etym type="sub">dfp</etym>
# <entry id="dfp"> זָבַח strong="2076" <etym type="main" root="זבח">
#
# main the entry IS a root. Its `root` attribute is the root; where
# BDB gives none (primitive nouns — יד, בן, דם) the word stands
# as its own root.
# sub the entry is filed under another; follow it up to its main.
# single BDB files the word alone, under no root (משה, שם, עיר). It is
# its own heading, and is engraved as such.
# (none) the index records no etymology at all — mostly proper names
# (יוסף, מצרים). No shoresh. The builder does not supply one.
This is a heuristic — not all words have triliteral roots (particles,
pronouns, etc). Returns None when derivation fails.
LEXICON_NS = "{http://openscriptures.github.com/morphhb/namespace}"
# Set by build_l1() before any verse is parsed.
_LEXICON: dict[str, tuple[str | None, str]] = {}
# How each word's shoresh was decided, and the content words left without.
_SHORESH_STATS: dict[str, int] = defaultdict(int)
_ROOTLESS: dict[tuple[str, str, str], int] = defaultdict(int)
def _hebrew_consonants(text: str | None) -> str:
"""Hebrew letters only — drops vowels, shin/sin dots, accents."""
return re.sub(r"[^\u05D0-\u05EA]", "", text or "")
def load_lexicon(lexicon_dir: str) -> dict[str, tuple[str | None, str]]:
"""
Read LexicalIndex.xml + AugIndex.xml into
augmented Strong's key ("1254a", "430") -> (root | None, kind)
where kind is "main" | "sub" | "single" | "none".
"""
entries: dict[str, dict] = {}
index = ET.parse(os.path.join(lexicon_dir, "LexicalIndex.xml")).getroot()
for e in index.iter(f"{LEXICON_NS}entry"):
w = e.find(f"{LEXICON_NS}w")
etym = e.find(f"{LEXICON_NS}etym")
entries[e.get("id")] = {
"word": _hebrew_consonants(w.text if w is not None else ""),
"type": etym.get("type") if etym is not None else None,
"root": _hebrew_consonants(etym.get("root")) if etym is not None else "",
"parent": (etym.text or "").strip() if etym is not None else "",
}
def root_of(entry_id: str) -> tuple[str | None, str]:
entry = entries[entry_id]
kind = entry["type"] or "none"
seen = {entry_id}
while entry["type"] == "sub" and entry["parent"] in entries and entry["parent"] not in seen:
seen.add(entry["parent"])
entry = entries[entry["parent"]]
if entry["type"] == "main":
return (entry["root"] or entry["word"]) or None, kind
if entry["type"] == "single":
return entry["word"] or None, kind
return None, kind
lexicon = {}
aug = ET.parse(os.path.join(lexicon_dir, "AugIndex.xml")).getroot()
for w in aug.iter(f"{LEXICON_NS}w"):
if w.text in entries:
lexicon[w.get("aug")] = root_of(w.text)
return lexicon
def lexicon_key_candidates(core_lemma: str) -> list[str]:
"""
morphhb writes "1254 a", "1187+", "7451"; the index is keyed "1254a",
"1187", and — where OSHB split a Strong's number — only "7451a",
"7451b"…. Try the exact key, then the lettered ones.
"""
key = re.sub(r"[\s+]", "", core_lemma)
if not key or not key[0].isdigit():
return []
if key[-1].isalpha():
return [key]
return [key] + [key + letter for letter in "abcdefgh"]
def shoresh_from_lexicon(core_lemma: str, morph_info: dict) -> tuple[str | None, str]:
"""
The word's root, read from the lexicon by its lemma.
Returns (shoresh | None, why) where why is one of
"main" "sub" "single" — a root was read
"none" — the lexicon records no etymology
"particle" — prepositions, conjunctions, particles and
pronouns are engraved without a shoresh,
as they always were (את, אשר, כי, הוא)
"unlisted" — the lemma is not in the lexical index
"""
pos = morph_info.get("pos", "")
# Particles, prepositions, conjunctions, pronouns don't have shoreshim
if pos in ("preposition", "particle", "conjunction", "pronoun", "suffix"):
return None
return None, "particle"
# For the consonantal text, we want only Hebrew letters
hebrew_only = re.sub(r"[^\u05D0-\u05EA]", "", consonants)
found = [_LEXICON[k] for k in lexicon_key_candidates(core_lemma) if k in _LEXICON]
if not found:
return None, "unlisted"
if len(hebrew_only) < 2:
return None
stem = morph_info.get("stem", "")
conjugation = morph_info.get("conjugation", "")
if pos == "verb":
# Strip imperfect/jussive/vav-consecutive prefixes: י מ נ ת א
# These are conjugation prefixes, not part of the root
if conjugation in ("imperfect", "vav_consecutive", "jussive"):
if len(hebrew_only) >= 4 and hebrew_only[0] in "ימנתא":
hebrew_only = hebrew_only[1:]
# Strip hifil/hofal he prefix
if stem in ("hifil", "hofal") and len(hebrew_only) >= 4:
if hebrew_only[0] == "ה":
hebrew_only = hebrew_only[1:]
# Imperfect hifil: prefix already stripped, check for yod remnant
# Strip hitpael tav prefix (and sometimes he)
elif stem == "hitpael" and len(hebrew_only) >= 4:
if hebrew_only[0] == "ה" and len(hebrew_only) >= 5 and hebrew_only[1] == "ת":
hebrew_only = hebrew_only[2:]
elif hebrew_only[0] == "ת":
hebrew_only = hebrew_only[1:]
# Strip nifal nun prefix
elif stem == "nifal" and len(hebrew_only) >= 4:
if hebrew_only[0] == "נ":
hebrew_only = hebrew_only[1:]
# Strip participle mem prefix
if conjugation in ("participle_active", "participle_passive"):
if len(hebrew_only) >= 4 and hebrew_only[0] == "מ":
hebrew_only = hebrew_only[1:]
elif pos == "noun":
# Strip common noun prefixes: מ (place/instrument nouns), ת (abstract nouns)
if len(hebrew_only) >= 4 and hebrew_only[0] in "מת":
pass # Don't strip by default for nouns — too aggressive
# Strip common suffixes
for suffix in ["ים", "ות", "ין", "תי", "תם", "תן", "נו"]:
if hebrew_only.endswith(suffix) and len(hebrew_only) - len(suffix) >= 2:
hebrew_only = hebrew_only[:-len(suffix)]
break
for suffix in ["ה", "ת", "ו", "י", "ך", "ם", "ן"]:
if hebrew_only.endswith(suffix) and len(hebrew_only) - 1 >= 3:
hebrew_only = hebrew_only[:-1]
break
if len(hebrew_only) == 3:
return hebrew_only
elif len(hebrew_only) == 2:
return hebrew_only
elif len(hebrew_only) > 3:
# Take first three as best guess
return hebrew_only[:3]
return hebrew_only if hebrew_only else None
# An unlettered lemma that OSHB split (7451 → 7451a, 7451b): accept
# the root only if every sense agrees on it.
roots = {root for root, _ in found}
if len(roots) > 1:
return None, "unlisted"
return found[0]
def collect_word_text(w_elem) -> tuple[str, list[dict], list[dict]]:
@@ -514,7 +573,6 @@ def parse_word(w_elem, position: int) -> dict:
# Extract consonantal text (strip everything but Hebrew letters)
consonants_full = strip_vowels_and_cantillation(joined_text)
consonants_core = strip_vowels_and_cantillation(core_text)
# Extract cantillation marks
teamim = extract_teamim(joined_text)
@@ -522,8 +580,11 @@ def parse_word(w_elem, position: int) -> dict:
# Parse morphology
morph_info = parse_morph_code(core_morph)
# Derive shoresh
shoresh = derive_shoresh_from_word(consonants_core, morph_info)
# Read the shoresh from the lexicon (tablet 34)
shoresh, why = shoresh_from_lexicon(core_lemma, morph_info)
_SHORESH_STATS[why] += 1
if shoresh is None and why in ("none", "unlisted"):
_ROOTLESS[(core_lemma, strip_vowels_and_cantillation(core_text), why)] += 1
# Build prefix string
prefix_str = "".join(prefixes) if prefixes else ""
@@ -574,16 +635,33 @@ def parse_book(xml_path: str, book_osis: str, book_hebrew: str) -> list[dict]:
return verses
def build_l1(morphhb_dir: str, output_dir: str):
def _source_commit(path: str) -> str:
"""The git commit a quarry stands at — so a rebuild can be repeated."""
import subprocess
try:
out = subprocess.run(
["git", "-C", path, "rev-parse", "HEAD"],
capture_output=True, text=True, check=True,
)
return out.stdout.strip()
except Exception:
return "unknown"
def build_l1(morphhb_dir: str, output_dir: str, lexicon_dir: str):
"""Build the complete L1 Torah filesystem."""
wlc_dir = os.path.join(morphhb_dir, "wlc")
torah_dir = os.path.join(output_dir, "torah")
_LEXICON.clear()
_LEXICON.update(load_lexicon(lexicon_dir))
_SHORESH_STATS.clear()
_ROOTLESS.clear()
total_verses = 0
total_words = 0
total_roots = set()
missing_teamim = []
failed_shoresh = []
book_stats = {}
print("=" * 60)
@@ -635,17 +713,6 @@ def build_l1(morphhb_dir: str, output_dir: str):
if word.get("shoresh"):
total_roots.add(word["shoresh"])
else:
# Only flag content words (not particles) as failed
morph_info = parse_morph_code(
word["morph"].split("/")[-1].lstrip("H") if word["morph"] else ""
)
if morph_info.get("pos") in ("verb", "noun", "adjective"):
failed_shoresh.append({
"ref": verse_data["ref"],
"word": word["text"],
"morph": word["morph"],
})
# Check for te'amim
verse_has_teamim = any(
@@ -708,7 +775,15 @@ def build_l1(morphhb_dir: str, output_dir: str):
print(f"Total words: {total_words}")
print(f"Unique shoreshim: {len(total_roots)}")
print(f"Verses missing te'amim: {len(missing_teamim)}")
print(f"Content words missing shoresh: {len(failed_shoresh)}")
words_with_shoresh = sum(_SHORESH_STATS[k] for k in ("main", "sub", "single"))
rootless = sorted(_ROOTLESS.items(), key=lambda kv: (-kv[1], kv[0]))
print(f"Words with a shoresh: {words_with_shoresh}")
print(f" read from a root entry (main): {_SHORESH_STATS['main']}")
print(f" filed under a root (sub): {_SHORESH_STATS['sub']}")
print(f" standing alone in the lexicon (single): {_SHORESH_STATS['single']}")
print(f"Particles, pronouns (no shoresh by rule): {_SHORESH_STATS['particle']}")
print(f"Lexicon records no etymology: {_SHORESH_STATS['none']}")
print(f"Lemma not in the lexical index: {_SHORESH_STATS['unlisted']}")
print(f"SHA-256: {sha256}")
print()
@@ -720,12 +795,10 @@ def build_l1(morphhb_dir: str, output_dir: str):
print(f" ... and {len(missing_teamim) - 20} more")
print()
if failed_shoresh:
print(f"Content words without shoresh ({len(failed_shoresh)}):")
for entry in failed_shoresh[:20]:
print(f" - {entry['ref']}: {entry['word']} ({entry['morph']})")
if len(failed_shoresh) > 20:
print(f" ... and {len(failed_shoresh) - 20} more")
if rootless:
print(f"Most frequent words the lexicon gives no root ({len(rootless)} distinct):")
for (lemma, word, why), n in rootless[:20]:
print(f" - {word} (lemma {lemma}, {why}) x{n}")
print()
# Write the report as JSON too
@@ -736,10 +809,25 @@ def build_l1(morphhb_dir: str, output_dir: str):
"total_words": total_words,
"unique_shoreshim": len(total_roots),
"verses_missing_teamim": len(missing_teamim),
"content_words_missing_shoresh": len(failed_shoresh),
"words_with_shoresh": words_with_shoresh,
"shoresh_provenance": {k: _SHORESH_STATS[k] for k in
("main", "sub", "single", "particle", "none", "unlisted")},
"sha256": sha256,
"missing_teamim_refs": missing_teamim,
"failed_shoresh_samples": failed_shoresh[:100],
"words_without_lexicon_root": [
{"lemma": lemma, "word": word, "why": why, "count": n}
for (lemma, word, why), n in rootless[:100]
],
"sources": {
"text": {
"name": "openscriptures/morphhb (Westminster Leningrad Codex)",
"commit": _source_commit(morphhb_dir),
},
"roots": {
"name": "openscriptures/HebrewLexicon (LexicalIndex.xml, AugIndex.xml — BDB roots)",
"commit": _source_commit(lexicon_dir),
},
},
}
report_path = os.path.join(output_dir, "build_report.json")
@@ -791,6 +879,25 @@ if __name__ == "__main__":
output_dir = repo_root / "l1"
# The root lexicon sits beside the text quarry.
# Override: SHORASHIM_LEXICON env var, or second CLI arg.
if len(sys.argv) > 2:
lexicon_dir = Path(sys.argv[2]).expanduser().resolve()
elif os.environ.get("SHORASHIM_LEXICON"):
lexicon_dir = Path(os.environ["SHORASHIM_LEXICON"]).expanduser().resolve()
else:
lexicon_dir = morphhb_dir.parent / "HebrewLexicon"
if not (lexicon_dir / "LexicalIndex.xml").exists():
print(f"ERROR: HebrewLexicon not found at {lexicon_dir}")
print()
print("The root lexicon (BDB roots by Strong's lemma) lives outside this repo.")
print("To fetch it:")
print(f" git clone https://github.com/openscriptures/HebrewLexicon.git {lexicon_dir}")
print()
print("Or pass a path as the second argument, or set SHORASHIM_LEXICON.")
sys.exit(1)
if not morphhb_dir.exists():
print(f"ERROR: morphhb not found at {morphhb_dir}")
print()
@@ -815,7 +922,7 @@ if __name__ == "__main__":
os.makedirs(output_dir, exist_ok=True)
report = build_l1(str(morphhb_dir), str(output_dir))
report = build_l1(str(morphhb_dir), str(output_dir), str(lexicon_dir))
# Verify counts
print()