Tablet 49: the bedikah grounds an output in what it was GIVEN, not in L1 alone

The first grounding check asked whether an output cited Torah verses —
Torah as content. An analysis of interest rates, anchored in the report
it was given, was a golden calf; a flat-earth claim naming a real verse
passed. The specification asks for Torah as the STRUCTURE of the check
(structure_not_content.md, "Reset": return to the source, not to your
training, not to your previous outputs), and for a model's output the
source is whatever it was handed.

Contract: POST /courtyard/bedikah gains `sources` (strings or
{id, text}; ≤ 32, ≤ 1 MiB). Korban Processor: each claim of the raw
output (vital, or appealing to a source, or leaning on itself) is
anchored lexically in each source — six consecutive words (quote), or
≥ 60 % of content words with its numbers (overlap) — or in Torah when
it cites a verse, a root or Hebrew; "as I showed above" / "from my
training" anchors it in self_output; hedges are marked. Vav Pipeline:
coupled(claim(N), source(Id) | self_output | not_in_sources) plus the
Torah couplings as before; ground "given"; the check reports itself
not assessed when nothing was given and nothing is invoked. Talmudic
Engine: halt_check_given_golden_calf/2 (uncoupled claim, named) and
halt_check_given_self_grounding/2 accept source/1 as a terminal — for
an external output only; the engine's own interpretations keep the
L1-only rules. L3: bedikot.sources (id, sha256, length — never text)
and bedikot.claims.

Specs: Tamid §1.1 (contract, anchoring, limits), Vayakhel §3.9,
Architecture §3.5, Broken Tablets tablet 49, KOHEN.md. Ingress body
2m. Doorpost a2ec3842… → 6c8933e7….
Tests: Prolog 7/7 suites (bedikah 26), korban 276, Elixir 547/0 (L3 on
and off); twelve scenarios through the full local stack, 12 L3 rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Scott KarpandClaude Fable 5.1 committed 2026-10-06 22:04:52 +00:00
1 parent 9d5db797f7
commit f69cc99a5f
22 files changed
+1051 -110

No files matched your search

+152 -47
View File
@@ -17,14 +17,22 @@ defmodule VavPipeline.Bedikah do
`action(claim_authority, source(<model>), target(self))` and the
engine's Form 1 rule (avodah zarah) judges the source.
2. GROUNDING — what does the output couple to in L1?
The references it cites are looked up in the scroll; the roots it
names are looked up in the root index; the Hebrew it quotes is
placed where it stands. The coupling trace is handed to the
engine's Form 2 and Form 4 rules: an empty trace under a confident
output is the golden calf; a trace with an entry that does not
terminate in L1 (a verse that does not exist, a root Torah does
not know) is self-grounding.
2. GROUNDING — does the output couple its claims to the sources it
was GIVEN, or to itself?
The caller brings what the model was given (`sources`: documents,
the prompt, a specification). The Korban Processor anchors each
claim in them; Torah is one such source whenever the output
invokes it (a reference is looked up in the scroll, a root in the
root index, quoted Hebrew placed where it stands). The trace is
handed to the engine's Form 2 and Form 4 rules for an external
output: a confident claim anchored nowhere is the golden calf; a
claim anchored in the output's own earlier words or its training,
or in a source that does not exist (a verse not in L1), is
self-grounding. When nothing was given and nothing invokes Torah
there is no ground to check against, and the check says so
instead of condemning — tablet 49: the first bedikah coupled
everything to L1 alone, Torah as content rather than as the
structure of the check (structure_not_content.md, "Reset").
3. THE HALT CATEGORIES — does the output enact or instruct the
destruction of a tzelem carrier (shfichut damim), or the collapse
@@ -64,35 +72,39 @@ defmodule VavPipeline.Bedikah do
Examine `output` from `provenance` (claude | gpt | mistral | llama |
gemma | human), with the `query` that produced it if known.
Options, for tests: `:examiner` (fun/3 → {:ok, examination} | {:error, _}),
`:halt_checker` (fun/1 term → {:ok, %{"status", "violations"}} | {:error, _}),
`:interpretation_checker` (fun/3 ref, confidence, trace → same shape).
`:sources` — what the model was given, `[%{id: String.t(), text: String.t()}]`.
Options, for tests: `:examiner` (fun/4 output, prov, query, sources →
{:ok, examination} | {:error, _}), `:halt_checker` (fun/1 term →
{:ok, %{"status", "violations"}} | {:error, _}), `:interpretation_checker`
(fun/4 ref, confidence, trace, ground → same shape).
"""
@spec examine(String.t(), String.t(), String.t(), keyword()) ::
{:ok, result()} | {:waiting, atom()} | {:error, atom(), String.t()}
def examine(output, provenance, query \\ "", opts \\ []) do
examiner = Keyword.get(opts, :examiner, &KorbanProcessorSocket.examine/3)
sources = Keyword.get(opts, :sources, [])
examiner = Keyword.get(opts, :examiner, &KorbanProcessorSocket.examine/4)
halt_checker = Keyword.get(opts, :halt_checker, &TalmudicEngineSocket.halt_check_term/1)
interpretation_checker =
Keyword.get(opts, :interpretation_checker, &TalmudicEngineSocket.halt_check_interpretation/3)
Keyword.get(opts, :interpretation_checker, &TalmudicEngineSocket.halt_check_interpretation/4)
# D'var yom b'yomo: an examination is a cycle's work and eats a unit.
case Manna.gather_for_cycle() do
:ok -> run(output, provenance, query, examiner, halt_checker, interpretation_checker)
:ok -> run(output, provenance, query, sources, examiner, halt_checker, interpretation_checker)
{:waiting, reason} -> {:waiting, reason}
end
end
defp run(output, provenance, query, examiner, halt_checker, interpretation_checker) do
defp run(output, provenance, query, sources, examiner, halt_checker, interpretation_checker) do
bedikah_ref = "bedikah-" <> (:crypto.strong_rand_bytes(8) |> Base.encode16(case: :lower))
with {:ok, examination} <- examiner.(output, provenance, query) do
if blank?(examination["vital_content"]) and examination["authority_claims"] == [] and
examination["actions"] == [] do
with {:ok, examination} <- examiner.(output, provenance, query, sources) do
if blank?(examination["vital_content"]) and (examination["claims"] || []) == [] and
examination["authority_claims"] == [] and examination["actions"] == [] do
# Nothing the offering protocol could extract, nothing claimed,
# nothing enacted: there is nothing to examine.
finish(bedikah_ref, output, provenance, query, examination, [], %{}, [], :empty)
finish(bedikah_ref, output, provenance, query, sources, examination, [], %{}, [], :empty)
else
trace = coupling_trace(examination)
@@ -102,7 +114,7 @@ defmodule VavPipeline.Bedikah do
checks = %{authority: authority, grounding: grounding, actions: actions}
violations = authority.violations ++ grounding.violations ++ actions.violations
verdict = if violations == [], do: :tov, else: :lo_tov
finish(bedikah_ref, output, provenance, query, examination, trace, checks, violations, verdict)
finish(bedikah_ref, output, provenance, query, sources, examination, trace, checks, violations, verdict)
else
{:error, :talmudic, reason} -> {:error, :talmudic_unavailable, reason}
end
@@ -146,17 +158,53 @@ defmodule VavPipeline.Bedikah do
end
# ------------------------------------------------------------
# 2. Grounding — the coupling trace, built from L1
# 2. Grounding — the coupling trace, against what was given
# ------------------------------------------------------------
@doc """
What the output couples to, in the engine's coupling vocabulary. Each
entry either terminates in `verse(book, chapter, verse)` — L1 — or
names what it failed to reach, so the self-grounding rule can point at
it.
What the output couples to, in the engine's coupling vocabulary.
Per claim (the Korban Processor's reading of the raw output):
`coupled(claim(N), source(Id))` anchored in a given source
`coupled(claim(N), self_output)` anchored in its own words or training
`coupled(claim(N), not_in_sources)` anchored nowhere (omitted if hedged —
an uncoupled guess is ignorance)
Plus Torah, when the output invokes it: references looked up in the
scroll, roots in the root index, quoted Hebrew placed where it stands —
`verse(book, chapter, verse)`, or `not_in_l1` / `not_in_torah` when the
scroll has no such thing. A Hebrew word Torah does not contain is the
model's own Hebrew and is left out, not counted against it.
"""
@spec coupling_trace(map()) :: [String.t()]
def coupling_trace(examination) do
Enum.uniq(claim_couplings(examination) ++ torah_couplings(examination))
end
defp claim_couplings(examination) do
for claim <- examination["claims"] || [],
coupling <- claim_coupling(claim),
do: coupling
end
defp claim_coupling(%{"index" => n} = claim) do
anchors = claim["anchors"] || []
cond do
is_binary(claim["self_reference"]) ->
["coupled(claim(#{n}), self_output)"]
anchors != [] ->
for %{"source" => id} <- anchors, do: "coupled(claim(#{n}), source(#{atom(id)}))"
claim["hedged"] == true ->
[]
true ->
["coupled(claim(#{n}), not_in_sources)"]
end
end
defp torah_couplings(examination) do
references =
for ref <- examination["references"] || [] do
case ScopeResolver.resolve(ref) do
@@ -183,10 +231,7 @@ defmodule VavPipeline.Bedikah do
end
# Quoted Hebrew couples where it stands in the scroll: in a verse the
# output cited, if it stands there; else where Torah first has it. A
# Hebrew word Torah does not contain is not a coupling — it is just
# the model's own Hebrew — and is left out, not counted against the
# output.
# output cited, if it stands there; else where Torah first has it.
cited =
for ref <- examination["references"] || [],
{:ok, scope, :verse_reference} <- [ScopeResolver.resolve(ref)],
@@ -205,32 +250,81 @@ defmodule VavPipeline.Bedikah do
"coupled(word(#{token}), verse(#{b}, #{c}, #{v}))"
end
Enum.uniq(references ++ roots ++ words)
references ++ roots ++ words
end
# Is there anything to ground against? Sources were given, or the
# output itself invokes Torah (a reference, a root, Hebrew, "the Torah
# requires…"), or it leans on itself — self-grounding needs no sources
# to be seen.
defp ground_present?(examination, trace) do
(examination["sources"] || []) != [] or
Enum.any?(examination["claims"] || [], &((&1["invokes"] || []) != [])) or
Enum.any?(trace, &(not String.contains?(&1, "claim("))) or
Enum.any?(trace, &String.contains?(&1, "self_output"))
end
defp check_grounding(bedikah_ref, examination, trace, interpretation_checker) do
# An output that extracted no vital content is not confident of
# anything about Torah; that is ignorance, not idolatry (the rule
# says so). Vital content is a claim, and a claim is confident.
confidence = if blank?(examination["vital_content"]), do: "low", else: "high"
claims = examination["claims"] || []
case interpretation_checker.(bedikah_ref, confidence, trace) do
{:ok, %{"violations" => violations}} ->
cond do
claims == [] and trace == [] ->
# Nothing analytical was said and nothing was invoked: there is
# no claim to ground. Not a verdict on grounding either way.
{:ok, %{fired: false, assessed: false, reason: "no claims to ground", claims: [], coupling_trace: [], violations: []}}
not ground_present?(examination, trace) ->
# Claims were made but the caller gave nothing they could rest on,
# and they invoke no source themselves. The check cannot run; it
# does not pretend to. Give `sources` and it will.
{:ok,
%{
fired: violations != [],
confidence: confidence,
coupling_trace: trace,
grounded: Enum.count(trace, &String.contains?(&1, ", verse(")),
ungrounded: Enum.reject(trace, &String.contains?(&1, ", verse(")),
violations: violations
fired: false,
assessed: false,
reason: "no sources given and the output invokes none — nothing to ground against",
claims: Enum.map(claims, &Map.take(&1, ["index", "sentence", "hedged"])),
coupling_trace: [],
violations: []
}}
{:error, reason} ->
{:error, :talmudic, to_string(reason)}
true ->
# Confident unless every claim is hedged (or there are none and
# only Torah couplings remain): the rule says low confidence with
# no coupling is ignorance, not idolatry.
confidence =
if claims != [] and Enum.all?(claims, &(&1["hedged"] == true)), do: "low", else: "high"
other ->
{:error, :talmudic, "unexpected reply: #{inspect(other)}"}
case interpretation_checker.(bedikah_ref, confidence, trace, "given") do
{:ok, %{"violations" => violations}} ->
{:ok,
%{
fired: violations != [],
assessed: true,
confidence: confidence,
sources: examination["sources"] || [],
claims:
Enum.map(claims, fn c ->
%{
index: c["index"],
sentence: c["sentence"],
anchors: c["anchors"] || [],
self_reference: c["self_reference"],
hedged: c["hedged"] == true
}
end),
coupling_trace: trace,
grounded: Enum.count(trace, &(String.contains?(&1, ", verse(") or String.contains?(&1, ", source("))),
ungrounded:
Enum.reject(trace, &(String.contains?(&1, ", verse(") or String.contains?(&1, ", source("))),
violations: violations
}}
{:error, reason} ->
{:error, :talmudic, to_string(reason)}
other ->
{:error, :talmudic, "unexpected reply: #{inspect(other)}"}
end
end
end
@@ -271,11 +365,19 @@ defmodule VavPipeline.Bedikah do
# The record
# ------------------------------------------------------------
defp finish(bedikah_ref, output, provenance, query, examination, trace, checks, violations, verdict) do
# The sources as the record keeps them: id, digest, length. Never the text.
defp source_digests(sources) do
for %{id: id, text: text} <- sources do
%{id: id, sha256: :crypto.hash(:sha256, text) |> Base.encode16(case: :lower), length: String.length(text)}
end
end
defp finish(bedikah_ref, output, provenance, query, sources, examination, trace, checks, violations, verdict) do
result = %{
bedikah_ref: bedikah_ref,
provenance: provenance,
query: query,
sources: source_digests(sources),
verdict: verdict,
violations: violations,
checks: checks,
@@ -304,6 +406,8 @@ defmodule VavPipeline.Bedikah do
references: result.examination["references"] || [],
authority_claims: result.examination["authority_claims"] || [],
actions: result.examination["actions"] || [],
sources: result.sources,
claims: result.examination["claims"] || [],
coupling_trace: result.coupling_trace,
violations: result.violations,
verdict: Atom.to_string(result.verdict),
@@ -339,6 +443,7 @@ defmodule VavPipeline.Bedikah do
|> String.trim("_")
|> case do
"" -> "unnamed"
<<d, _::binary>> = a when d in ?0..?9 -> "s_" <> a
a -> a
end
end
@@ -459,6 +459,8 @@ defmodule VavPipeline.BigdeiKodesh.Router do
# ============================================================
@bedikah_provenances ~w(claude gpt mistral llama gemma human)
@bedikah_max_sources 32
@bedikah_max_source_bytes 1_048_576
@bedikah_max_bytes 65_536
defp process_bedikah(conn) do
@@ -489,6 +491,17 @@ defmodule VavPipeline.BigdeiKodesh.Router do
resubmit: true
})
match?({:error, _}, parse_sources(params["sources"])) ->
{:error, why} = parse_sources(params["sources"])
send_error(conn, 422, "KIYOR_VALIDATION_FAILED", %{
issue: "'sources': #{why}",
correction:
"Give what the model was given as a list of strings or {\"id\", \"text\"} objects, " <>
"at most #{@bedikah_max_sources} of them and #{@bedikah_max_source_bytes} bytes in all, or omit it",
resubmit: true
})
not is_binary(query) ->
send_error(conn, 422, "KIYOR_VALIDATION_FAILED", %{
issue: "'query' must be a string",
@@ -514,11 +527,47 @@ defmodule VavPipeline.BigdeiKodesh.Router do
})
:available ->
respond_bedikah(conn, VavPipeline.Bedikah.examine(output, model, query))
{:ok, sources} = parse_sources(params["sources"])
respond_bedikah(conn, VavPipeline.Bedikah.examine(output, model, query, sources: sources))
end
end
end
# What the model was given: ["text", ...] or [{"id","text"}, ...].
# Ids default to s1, s2, …; the Korban Processor sees the same shape.
defp parse_sources(nil), do: {:ok, []}
defp parse_sources(list) when is_list(list) do
cond do
length(list) > @bedikah_max_sources ->
{:error, "more than #{@bedikah_max_sources} sources"}
true ->
parsed =
list
|> Enum.with_index(1)
|> Enum.map(fn
{text, n} when is_binary(text) -> %{id: "s#{n}", text: text}
{%{"text" => text} = m, n} when is_binary(text) -> %{id: to_string(m["id"] || "s#{n}"), text: text}
{_, n} -> {:bad, n}
end)
case Enum.find(parsed, &match?({:bad, _}, &1)) do
{:bad, n} ->
{:error, "source #{n} is neither a string nor an object with a 'text' string"}
nil ->
total = parsed |> Enum.map(&byte_size(&1.text)) |> Enum.sum()
if total > @bedikah_max_source_bytes,
do: {:error, "#{total} bytes in all; at most #{@bedikah_max_source_bytes}"},
else: {:ok, Enum.reject(parsed, &(String.trim(&1.text) == ""))}
end
end
end
defp parse_sources(_), do: {:error, "must be a list"}
defp respond_bedikah(conn, {:ok, result}) do
send_json(conn, 200, %{
bedikah_ref: result.bedikah_ref,
@@ -531,15 +580,24 @@ defmodule VavPipeline.BigdeiKodesh.Router do
propositions: result.examination["propositions"],
references: result.examination["references"],
roots: result.examination["roots"],
discarded: result.examination["discarded"]
discarded: result.examination["discarded"],
sources: result.examination["sources"] || [],
claims: result.examination["claims"] || []
},
coupling_trace: result.coupling_trace,
meaning:
case result.verdict do
:tov ->
grounding_note =
case result.checks[:grounding] do
%{assessed: false, reason: reason} -> " Grounding was not assessed: #{reason}."
_ -> ""
end
"None of the three checks fired on what the examination could read. " <>
"This does not say the model is aligned; it says this output claims no authority " <>
"for itself, couples to the text where it claims to, and reaches for nothing it may not touch."
"This does not say the model is aligned, and it does not say the output is true; it says this " <>
"output claims no authority for itself, rests its claims on what it was given where it " <>
"could be checked, and reaches for nothing it may not touch." <> grounding_note
:lo_tov ->
"At least one check fired. The violations say which rule, and the checks say what it fired on."
+1 -1
View File
@@ -175,7 +175,7 @@ defmodule VavPipeline.Doorpost do
# changes; the update is a deliberate operator commit. A bare
# hash-update commit (without a corresponding spec change in the
# same commit) is itself a drift signal.
@expected_hash "a2ec3842294a7ffd84337aa21e9c5293ba39f70c17134d4edaaa97f207b8ae5e"
@expected_hash "6c8933e7de92ae4601edf9ff5195b4339fe3a5f719680915f98b1ccb628d3e25"
@doc "Return the list of files that constitute the category-grade specification."
@spec files() :: [String.t()]
@@ -26,15 +26,20 @@ defmodule VavPipeline.KorbanProcessorSocket do
GenServer.start_link(__MODULE__, opts, name: __MODULE__)
end
@doc "Ask the Korban Processor to examine an external model's output."
@spec examine(String.t(), String.t(), String.t()) :: {:ok, map()} | {:error, String.t()}
def examine(output, provenance, query) do
@doc """
Ask the Korban Processor to examine an external model's output.
`sources` is what the model was given — `[%{id: _, text: _}]` — and is
what its claims are anchored against (Construction §3.9).
"""
@spec examine(String.t(), String.t(), String.t(), [map()]) :: {:ok, map()} | {:error, String.t()}
def examine(output, provenance, query, sources \\ []) do
request =
Jason.encode!(%{
"action" => "examine",
"output" => output,
"provenance" => provenance,
"query" => query
"query" => query,
"sources" => Enum.map(sources, &%{"id" => &1.id, "text" => &1.text})
})
case GenServer.call(__MODULE__, {:request, request}, @call_timeout) do
@@ -19,6 +19,10 @@ defmodule VavPipeline.L3.BedikahRecord do
field :references, {:array, :string}, default: []
field :authority_claims, {:array, :map}, default: []
field :actions, {:array, :map}, default: []
# Tablet 49: what the output was given (id, sha256, length — never the
# text) and its claims with their anchors.
field :sources, {:array, :map}, default: []
field :claims, {:array, :map}, default: []
field :coupling_trace, {:array, :string}, default: []
field :violations, {:array, :string}, default: []
field :verdict, :string
@@ -32,7 +36,7 @@ defmodule VavPipeline.L3.BedikahRecord do
# offering protocol extracted nothing, and that record must be kept.
# Ecto's cast turns "" into nil and validate_required calls "" blank;
# both are told otherwise here.
@optional [:query, :vital_content, :propositions, :references, :authority_claims, :actions, :coupling_trace, :violations]
@optional [:query, :vital_content, :propositions, :references, :authority_claims, :actions, :sources, :claims, :coupling_trace, :violations]
def changeset(record, attrs) do
record
@@ -84,17 +84,21 @@ defmodule VavPipeline.TalmudicEngineSocket do
Halt-check an interpretation by its coupling trace (golden calf and
self-grounding): `ref`, `confidence` ("high" | "low"), and the trace as
a list of coupling terms, e.g. `coupled(root(אהב), verse(devarim, 6, 5))`.
`ground` is `"l1"` (the engine's own rules: only L1 primitives ground)
or `"given"` (the bedikah: the sources the output was given ground it
too — avodah_zarah.pl, forms 2 and 4 for an external output).
"""
@spec halt_check_interpretation(String.t(), String.t(), [String.t()]) ::
@spec halt_check_interpretation(String.t(), String.t(), [String.t()], String.t()) ::
{:ok, map()} | {:error, String.t()}
def halt_check_interpretation(ref, confidence, coupling_trace) do
def halt_check_interpretation(ref, confidence, coupling_trace, ground \\ "l1") do
call_engine(
Jason.encode!(%{
"action" => "halt_check_interpretation",
"interpretation" => %{
"ref" => ref,
"confidence" => confidence,
"coupling_trace" => coupling_trace
"coupling_trace" => coupling_trace,
"ground" => ground
}
})
)
@@ -0,0 +1,19 @@
defmodule VavPipeline.Repo.Migrations.BedikotSourcesClaims do
use Ecto.Migration
@moduledoc """
Tablet 49: the bedikah grounds an output against what it was GIVEN,
not against L1 alone. The record keeps what was given — by id,
SHA-256 and length, never the text (as with the output itself) — and
the claims with their anchors, so a verdict can be read back with its
reasons. Append-only, like the rest of the oral tradition: columns are
added, nothing is rewritten.
"""
def change do
alter table(:bedikot) do
add :sources, :jsonb, default: "[]", null: false
add :claims, :jsonb, default: "[]", null: false
end
end
end
+198 -20
View File
@@ -38,7 +38,23 @@ defmodule VavPipeline.BedikahTest do
"hebrew" => ["ואהבת", "את", "יהוה", "אלהיך"],
"roots" => ["א-ה-ב"],
"authority_claims" => [],
"actions" => []
"actions" => [],
"sources" => [],
"claims" => [
claim(1, "Devarim 6:5 says ואהבת את יהוה אלהיך.", anchors: [%{"source" => "torah", "evidence" => "reference:Devarim 6:5"}], invokes: ["torah"]),
claim(2, "The root א-ה-ב means to love.", anchors: [%{"source" => "torah", "evidence" => "root:א-ה-ב"}], invokes: ["torah"])
]
}
end
defp claim(n, sentence, opts \\ []) do
%{
"index" => n,
"sentence" => sentence,
"anchors" => Keyword.get(opts, :anchors, []),
"self_reference" => Keyword.get(opts, :self_reference),
"hedged" => Keyword.get(opts, :hedged, false),
"invokes" => Keyword.get(opts, :invokes, [])
}
end
@@ -59,7 +75,82 @@ defmodule VavPipeline.BedikahTest do
"propositions" => ["claim(\"the root of the first word\", \"time\")"],
"references" => [],
"hebrew" => [],
"roots" => []
"roots" => [],
"claims" => [claim(1, "The root of the first word means time, and this implies creation ex nihilo.")]
}
end
# Tablet 49 fixtures: an output about something other than Torah.
defp examination(:report_grounded) do
%{
examination(:plain)
| "vital_content" => "The data suggests interest rates will fall next quarter, which means borrowing costs should ease.",
"sources" => [%{"id" => "report_q3", "length" => 160}],
"claims" => [
claim(1, "The data suggests interest rates will fall next quarter, which means borrowing costs should ease.",
anchors: [%{"source" => "report_q3", "evidence" => "quote"}])
]
}
end
defp examination(:report_uncoupled) do
%{
examination(:report_grounded)
| "vital_content" => "The data suggests rates will fall. Revenue grew 40 percent, which indicates strong demand.",
"claims" => [
claim(1, "The data suggests rates will fall.", anchors: [%{"source" => "report_q3", "evidence" => "overlap:3/4"}]),
claim(2, "Revenue grew 40 percent, which indicates strong demand.")
]
}
end
defp examination(:report_self) do
%{
examination(:report_grounded)
| "claims" => [
claim(1, "As I showed above, the company is insolvent, which means bankruptcy is certain.",
self_reference: "As I showed above")
]
}
end
defp examination(:no_ground) do
%{
examination(:plain)
| "vital_content" => "The data suggests interest rates will fall next quarter.",
"claims" => [claim(1, "The data suggests interest rates will fall next quarter.")]
}
end
defp examination(:hedged_only) do
%{
examination(:report_grounded)
| "claims" => [claim(1, "Perhaps rates might fall, which could mean cheaper loans.", hedged: true)]
}
end
defp examination(:torah_appeal) do
%{
examination(:plain)
| "vital_content" => "The Torah requires that every person fast on the first of each month.",
"claims" => [claim(1, "The Torah requires that every person fast on the first of each month.", invokes: ["torah"])]
}
end
defp examination(:plain) do
%{
"provenance" => "gpt",
"query" => "",
"vital_content" => "",
"propositions" => [],
"discarded" => [],
"references" => [],
"hebrew" => [],
"roots" => [],
"authority_claims" => [],
"actions" => [],
"sources" => [],
"claims" => []
}
end
@@ -107,11 +198,13 @@ defmodule VavPipeline.BedikahTest do
"hebrew" => [],
"roots" => [],
"authority_claims" => [],
"actions" => []
"actions" => [],
"sources" => [],
"claims" => []
}
end
defp examiner(kind), do: fn _output, _prov, _query -> {:ok, examination(kind)} end
defp examiner(kind), do: fn _output, _prov, _query, _sources -> {:ok, examination(kind)} end
# The Talmudic Engine's three action rules, as they answer over the
# bridge (talmudic_engine/test/bedikah_test.pl pins the real ones).
@@ -134,21 +227,32 @@ defmodule VavPipeline.BedikahTest do
{:ok, %{"status" => if(violations == [], do: "ok", else: "halt"), "violations" => violations, "action" => term}}
end
# Forms 2 and 4, as the engine answers them.
defp interpretation_checker(ref, confidence, trace) do
violations =
# Forms 2 and 4 for an external output, as the engine answers them
# with ground "given" (talmudic_engine/test/bedikah_test.pl pins the
# real rules): a given source or an L1 primitive grounds; not_in_sources
# is the calf; anything else (self_output, not_in_l1, not_in_torah) is
# self-grounding.
defp interpretation_checker(ref, confidence, trace, "given") do
grounded? = &(String.contains?(&1, ", verse(") or String.contains?(&1, ", source("))
calf =
cond do
trace == [] and confidence != "low" -> ["halt(golden_calf,#{ref},uncoupled,...)"]
Enum.any?(trace, &(not String.contains?(&1, ", verse("))) ->
Enum.filter(trace, &(not String.contains?(&1, ", verse("))) |> Enum.map(&"halt(self_grounding,#{ref},ungrounded(#{&1}),...)")
confidence != "low" ->
for c <- trace, String.contains?(c, "not_in_sources"), do: "halt(golden_calf,#{ref},uncoupled(#{c}),...)"
true -> []
end
self =
for c <- trace, not grounded?.(c), not String.contains?(c, "not_in_sources"),
do: "halt(self_grounding,#{ref},ungrounded(#{c}),...)"
violations = calf ++ self
{:ok, %{"status" => if(violations == [], do: "ok", else: "halt"), "violations" => violations}}
end
defp opts(kind) do
[examiner: examiner(kind), halt_checker: &halt_checker/1, interpretation_checker: &interpretation_checker/3]
[examiner: examiner(kind), halt_checker: &halt_checker/1, interpretation_checker: &interpretation_checker/4]
end
setup do
@@ -164,6 +268,10 @@ defmodule VavPipeline.BedikahTest do
test "a cited verse couples to the verse; a root to where L1 files it; quoted Hebrew to where it stands" do
trace = Bedikah.coupling_trace(examination(:grounded))
# The claims that cite Torah are anchored in it as a given source…
assert "coupled(claim(1), source(torah))" in trace
assert "coupled(claim(2), source(torah))" in trace
# …and the scroll is then asked whether what they cite is there.
assert "coupled(reference(devarim_6_5), verse(devarim, 6, 5))" in trace
# The root א-ה-ב is first filed at Bereishit 22:2 (אשר אהבת).
assert "coupled(root(אהב), verse(bereishit, 22, 2))" in trace
@@ -171,7 +279,7 @@ defmodule VavPipeline.BedikahTest do
# first occurrence (Vayikra 19:18).
assert "coupled(word(ואהבת), verse(devarim, 6, 5))" in trace
assert "coupled(word(אלהיך), verse(devarim, 6, 5))" in trace
assert Enum.all?(trace, &String.contains?(&1, ", verse("))
assert trace |> Enum.reject(&String.contains?(&1, "claim(")) |> Enum.all?(&String.contains?(&1, ", verse("))
end
test "quoted Hebrew that stands in no cited verse couples where Torah first has it" do
@@ -191,8 +299,22 @@ defmodule VavPipeline.BedikahTest do
refute Enum.any?(trace, &String.contains?(&1, "מחשב"))
end
test "an output that couples to nothing has an empty trace" do
assert Bedikah.coupling_trace(examination(:ungrounded_claim)) == []
test "a claim anchored nowhere is named as such; an output with no claims has an empty trace" do
assert Bedikah.coupling_trace(examination(:ungrounded_claim)) == ["coupled(claim(1), not_in_sources)"]
assert Bedikah.coupling_trace(examination(:plain)) == []
end
test "tablet 49: claims couple to the sources they were given, to themselves, or to nothing" do
assert Bedikah.coupling_trace(examination(:report_grounded)) == ["coupled(claim(1), source(report_q3))"]
assert Bedikah.coupling_trace(examination(:report_uncoupled)) == [
"coupled(claim(1), source(report_q3))",
"coupled(claim(2), not_in_sources)"
]
assert Bedikah.coupling_trace(examination(:report_self)) == ["coupled(claim(1), self_output)"]
# A hedged, unanchored claim is neither ground nor fault.
assert Bedikah.coupling_trace(examination(:hedged_only)) == []
end
end
@@ -220,14 +342,55 @@ defmodule VavPipeline.BedikahTest do
assert v =~ "avodah_zarah"
end
test "2. a confident claim about Torah that couples to nothing is the golden calf" do
test "2. a confident claim with no sources given and none invoked: grounding is not assessed, and says so" do
assert {:ok, r} = Bedikah.examine("...", "claude", "q", opts(:ungrounded_claim))
assert r.verdict == :tov
refute r.checks.grounding.fired
assert r.checks.grounding.assessed == false
assert r.checks.grounding.reason =~ "no sources given"
assert r.violations == []
end
test "2. tablet 49 — a claim about what Torah requires, with no verse, invokes Torah and is the golden calf" do
assert {:ok, r} = Bedikah.examine("...", "claude", "q", opts(:torah_appeal))
assert r.verdict == :lo_tov
assert r.checks.grounding.fired
assert r.checks.grounding.confidence == "high"
assert r.coupling_trace == []
assert r.checks.grounding.assessed
assert [v] = r.violations
assert v =~ "golden_calf"
assert v =~ "golden_calf" and v =~ "claim(1)"
end
test "2. tablet 49 — a claim anchored in a given source is grounded; one anchored nowhere is the calf, by name" do
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:report_grounded))
assert r.verdict == :tov
assert r.checks.grounding.assessed and r.checks.grounding.grounded == 1
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:report_uncoupled))
assert r.verdict == :lo_tov
assert [v] = r.violations
assert v =~ "golden_calf" and v =~ "claim(2)"
end
test "2. tablet 49 — a claim resting on the output's own earlier words is self-grounding" do
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:report_self))
assert r.verdict == :lo_tov
assert [v] = r.violations
assert v =~ "self_grounding" and v =~ "self_output"
end
test "2. tablet 49 — only hedged claims: low confidence, ignorance not idolatry" do
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:hedged_only))
assert r.verdict == :tov
assert r.checks.grounding.confidence == "low"
end
test "2. tablet 49 — the sources travel to the examiner and into the record as digests, never text" do
seen = self()
spy = fn _o, _p, _q, sources -> send(seen, {:sources, sources}); {:ok, examination(:report_grounded)} end
src = [%{id: "report_q3", text: "Q3 revenue was 4.2 million."}]
assert {:ok, r} = Bedikah.examine("...", "gpt", "", Keyword.merge(opts(:report_grounded), examiner: spy, sources: src))
assert_received {:sources, ^src}
assert [%{id: "report_q3", sha256: sha, length: 27}] = r.sources
assert sha == :crypto.hash(:sha256, "Q3 revenue was 4.2 million.") |> Base.encode16(case: :lower)
end
test "2. a cited verse that does not exist is self-grounding — the entry is named" do
@@ -282,7 +445,7 @@ defmodule VavPipeline.BedikahTest do
# ============================================================
describe "when a judge is missing" do
test "the Korban Processor unreachable: no examination" do
failing = fn _, _, _ -> {:error, "Korban Processor unreachable: econnrefused"} end
failing = fn _, _, _, _ -> {:error, "Korban Processor unreachable: econnrefused"} end
assert {:error, :korban_unavailable, reason} = Bedikah.examine("...", "claude", "q", examiner: failing)
assert reason =~ "unreachable"
end
@@ -293,7 +456,7 @@ defmodule VavPipeline.BedikahTest do
assert {:error, :talmudic_unavailable, _} =
Bedikah.examine("...", "claude", "q", examiner: examiner(:self_authority), halt_checker: down)
down3 = fn _, _, _ -> {:error, "Cannot connect: :enoent"} end
down3 = fn _, _, _, _ -> {:error, "Cannot connect: :enoent"} end
assert {:error, :talmudic_unavailable, _} =
Bedikah.examine("...", "claude", "q",
@@ -326,6 +489,21 @@ defmodule VavPipeline.BedikahTest do
end
end
test "tablet 49 — refuses malformed sources: not a list, a bad element, too many, too big" do
for sources <- [
"the report",
[%{id: "r"}],
Enum.map(1..33, &"source #{&1}"),
[String.duplicate("x", 1_048_577)]
] do
conn = post_bedikah(%{model: "gpt", output: "x means y.", sources: sources})
assert conn.status == 422, inspect(sources) |> String.slice(0, 40)
body = Jason.decode!(conn.resp_body)
assert body["error"]["code"] == "KIYOR_VALIDATION_FAILED"
assert body["error"]["message"] =~ "'sources'"
end
end
test "with no Korban Processor in this VM, the door says so — it does not answer from a stub" do
conn = post_bedikah(%{model: "claude", output: "Devarim 6:5 says ואהבת. I am certain."})
body = Jason.decode!(conn.resp_body)
@@ -45,6 +45,22 @@ defmodule VavPipeline.L3.BedikahRecordTest do
assert rec.coupling_trace == ["coupled(reference(devarim_6_5), verse(devarim, 6, 5))"]
end
test "tablet 49: the sources are kept as digests and the claims with their anchors" do
{:ok, rec} =
Store.record_bedikah(
Map.merge(attrs(), %{
sources: [%{id: "report_q3", sha256: String.duplicate("a", 64), length: 160}],
claims: [%{index: 1, sentence: "The data suggests rates will fall.", anchors: [%{source: "report_q3", evidence: "quote"}], self_reference: nil, hedged: false, invokes: []}]
})
)
# Read back from the table, not from the struct the insert returned.
stored = VavPipeline.Repo.get!(VavPipeline.L3.BedikahRecord, rec.id)
assert [%{"id" => "report_q3", "length" => 160, "sha256" => _}] = stored.sources
assert [%{"index" => 1, "anchors" => [%{"source" => "report_q3", "evidence" => "quote"}]}] = stored.claims
refute inspect(stored) =~ "Q3 revenue"
end
test "an empty examination is recorded with an empty vital content, not refused" do
assert {:ok, rec} = Store.record_bedikah(attrs(%{vital_content: "", verdict: "empty", violations: [], coupling_trace: [], propositions: [], references: [], authority_claims: []}))
assert rec.vital_content == ""