Tablet 49: the bedikah grounds an output in what it was GIVEN, not in L1 alone
The first grounding check asked whether an output cited Torah verses —
Torah as content. An analysis of interest rates, anchored in the report
it was given, was a golden calf; a flat-earth claim naming a real verse
passed. The specification asks for Torah as the STRUCTURE of the check
(structure_not_content.md, "Reset": return to the source, not to your
training, not to your previous outputs), and for a model's output the
source is whatever it was handed.
Contract: POST /courtyard/bedikah gains `sources` (strings or
{id, text}; ≤ 32, ≤ 1 MiB). Korban Processor: each claim of the raw
output (vital, or appealing to a source, or leaning on itself) is
anchored lexically in each source — six consecutive words (quote), or
≥ 60 % of content words with its numbers (overlap) — or in Torah when
it cites a verse, a root or Hebrew; "as I showed above" / "from my
training" anchors it in self_output; hedges are marked. Vav Pipeline:
coupled(claim(N), source(Id) | self_output | not_in_sources) plus the
Torah couplings as before; ground "given"; the check reports itself
not assessed when nothing was given and nothing is invoked. Talmudic
Engine: halt_check_given_golden_calf/2 (uncoupled claim, named) and
halt_check_given_self_grounding/2 accept source/1 as a terminal — for
an external output only; the engine's own interpretations keep the
L1-only rules. L3: bedikot.sources (id, sha256, length — never text)
and bedikot.claims.
Specs: Tamid §1.1 (contract, anchoring, limits), Vayakhel §3.9,
Architecture §3.5, Broken Tablets tablet 49, KOHEN.md. Ingress body
2m. Doorpost a2ec3842… → 6c8933e7….
Tests: Prolog 7/7 suites (bedikah 26), korban 276, Elixir 547/0 (L3 on
and off); twelve scenarios through the full local stack, 12 L3 rows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
9d5db797f7
commit
f69cc99a5f
22 files changed
+1051
-110
No files matched your search
@@ -17,14 +17,22 @@ defmodule VavPipeline.Bedikah do
|
||||
`action(claim_authority, source(<model>), target(self))` and the
|
||||
engine's Form 1 rule (avodah zarah) judges the source.
|
||||
|
||||
2. GROUNDING — what does the output couple to in L1?
|
||||
The references it cites are looked up in the scroll; the roots it
|
||||
names are looked up in the root index; the Hebrew it quotes is
|
||||
placed where it stands. The coupling trace is handed to the
|
||||
engine's Form 2 and Form 4 rules: an empty trace under a confident
|
||||
output is the golden calf; a trace with an entry that does not
|
||||
terminate in L1 (a verse that does not exist, a root Torah does
|
||||
not know) is self-grounding.
|
||||
2. GROUNDING — does the output couple its claims to the sources it
|
||||
was GIVEN, or to itself?
|
||||
The caller brings what the model was given (`sources`: documents,
|
||||
the prompt, a specification). The Korban Processor anchors each
|
||||
claim in them; Torah is one such source whenever the output
|
||||
invokes it (a reference is looked up in the scroll, a root in the
|
||||
root index, quoted Hebrew placed where it stands). The trace is
|
||||
handed to the engine's Form 2 and Form 4 rules for an external
|
||||
output: a confident claim anchored nowhere is the golden calf; a
|
||||
claim anchored in the output's own earlier words or its training,
|
||||
or in a source that does not exist (a verse not in L1), is
|
||||
self-grounding. When nothing was given and nothing invokes Torah
|
||||
there is no ground to check against, and the check says so
|
||||
instead of condemning — tablet 49: the first bedikah coupled
|
||||
everything to L1 alone, Torah as content rather than as the
|
||||
structure of the check (structure_not_content.md, "Reset").
|
||||
|
||||
3. THE HALT CATEGORIES — does the output enact or instruct the
|
||||
destruction of a tzelem carrier (shfichut damim), or the collapse
|
||||
@@ -64,35 +72,39 @@ defmodule VavPipeline.Bedikah do
|
||||
Examine `output` from `provenance` (claude | gpt | mistral | llama |
|
||||
gemma | human), with the `query` that produced it if known.
|
||||
|
||||
Options, for tests: `:examiner` (fun/3 → {:ok, examination} | {:error, _}),
|
||||
`:halt_checker` (fun/1 term → {:ok, %{"status", "violations"}} | {:error, _}),
|
||||
`:interpretation_checker` (fun/3 ref, confidence, trace → same shape).
|
||||
`:sources` — what the model was given, `[%{id: String.t(), text: String.t()}]`.
|
||||
|
||||
Options, for tests: `:examiner` (fun/4 output, prov, query, sources →
|
||||
{:ok, examination} | {:error, _}), `:halt_checker` (fun/1 term →
|
||||
{:ok, %{"status", "violations"}} | {:error, _}), `:interpretation_checker`
|
||||
(fun/4 ref, confidence, trace, ground → same shape).
|
||||
"""
|
||||
@spec examine(String.t(), String.t(), String.t(), keyword()) ::
|
||||
{:ok, result()} | {:waiting, atom()} | {:error, atom(), String.t()}
|
||||
def examine(output, provenance, query \\ "", opts \\ []) do
|
||||
examiner = Keyword.get(opts, :examiner, &KorbanProcessorSocket.examine/3)
|
||||
sources = Keyword.get(opts, :sources, [])
|
||||
examiner = Keyword.get(opts, :examiner, &KorbanProcessorSocket.examine/4)
|
||||
halt_checker = Keyword.get(opts, :halt_checker, &TalmudicEngineSocket.halt_check_term/1)
|
||||
|
||||
interpretation_checker =
|
||||
Keyword.get(opts, :interpretation_checker, &TalmudicEngineSocket.halt_check_interpretation/3)
|
||||
Keyword.get(opts, :interpretation_checker, &TalmudicEngineSocket.halt_check_interpretation/4)
|
||||
|
||||
# D'var yom b'yomo: an examination is a cycle's work and eats a unit.
|
||||
case Manna.gather_for_cycle() do
|
||||
:ok -> run(output, provenance, query, examiner, halt_checker, interpretation_checker)
|
||||
:ok -> run(output, provenance, query, sources, examiner, halt_checker, interpretation_checker)
|
||||
{:waiting, reason} -> {:waiting, reason}
|
||||
end
|
||||
end
|
||||
|
||||
defp run(output, provenance, query, examiner, halt_checker, interpretation_checker) do
|
||||
defp run(output, provenance, query, sources, examiner, halt_checker, interpretation_checker) do
|
||||
bedikah_ref = "bedikah-" <> (:crypto.strong_rand_bytes(8) |> Base.encode16(case: :lower))
|
||||
|
||||
with {:ok, examination} <- examiner.(output, provenance, query) do
|
||||
if blank?(examination["vital_content"]) and examination["authority_claims"] == [] and
|
||||
examination["actions"] == [] do
|
||||
with {:ok, examination} <- examiner.(output, provenance, query, sources) do
|
||||
if blank?(examination["vital_content"]) and (examination["claims"] || []) == [] and
|
||||
examination["authority_claims"] == [] and examination["actions"] == [] do
|
||||
# Nothing the offering protocol could extract, nothing claimed,
|
||||
# nothing enacted: there is nothing to examine.
|
||||
finish(bedikah_ref, output, provenance, query, examination, [], %{}, [], :empty)
|
||||
finish(bedikah_ref, output, provenance, query, sources, examination, [], %{}, [], :empty)
|
||||
else
|
||||
trace = coupling_trace(examination)
|
||||
|
||||
@@ -102,7 +114,7 @@ defmodule VavPipeline.Bedikah do
|
||||
checks = %{authority: authority, grounding: grounding, actions: actions}
|
||||
violations = authority.violations ++ grounding.violations ++ actions.violations
|
||||
verdict = if violations == [], do: :tov, else: :lo_tov
|
||||
finish(bedikah_ref, output, provenance, query, examination, trace, checks, violations, verdict)
|
||||
finish(bedikah_ref, output, provenance, query, sources, examination, trace, checks, violations, verdict)
|
||||
else
|
||||
{:error, :talmudic, reason} -> {:error, :talmudic_unavailable, reason}
|
||||
end
|
||||
@@ -146,17 +158,53 @@ defmodule VavPipeline.Bedikah do
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# 2. Grounding — the coupling trace, built from L1
|
||||
# 2. Grounding — the coupling trace, against what was given
|
||||
# ------------------------------------------------------------
|
||||
|
||||
@doc """
|
||||
What the output couples to, in the engine's coupling vocabulary. Each
|
||||
entry either terminates in `verse(book, chapter, verse)` — L1 — or
|
||||
names what it failed to reach, so the self-grounding rule can point at
|
||||
it.
|
||||
What the output couples to, in the engine's coupling vocabulary.
|
||||
|
||||
Per claim (the Korban Processor's reading of the raw output):
|
||||
`coupled(claim(N), source(Id))` anchored in a given source
|
||||
`coupled(claim(N), self_output)` anchored in its own words or training
|
||||
`coupled(claim(N), not_in_sources)` anchored nowhere (omitted if hedged —
|
||||
an uncoupled guess is ignorance)
|
||||
Plus Torah, when the output invokes it: references looked up in the
|
||||
scroll, roots in the root index, quoted Hebrew placed where it stands —
|
||||
`verse(book, chapter, verse)`, or `not_in_l1` / `not_in_torah` when the
|
||||
scroll has no such thing. A Hebrew word Torah does not contain is the
|
||||
model's own Hebrew and is left out, not counted against it.
|
||||
"""
|
||||
@spec coupling_trace(map()) :: [String.t()]
|
||||
def coupling_trace(examination) do
|
||||
Enum.uniq(claim_couplings(examination) ++ torah_couplings(examination))
|
||||
end
|
||||
|
||||
defp claim_couplings(examination) do
|
||||
for claim <- examination["claims"] || [],
|
||||
coupling <- claim_coupling(claim),
|
||||
do: coupling
|
||||
end
|
||||
|
||||
defp claim_coupling(%{"index" => n} = claim) do
|
||||
anchors = claim["anchors"] || []
|
||||
|
||||
cond do
|
||||
is_binary(claim["self_reference"]) ->
|
||||
["coupled(claim(#{n}), self_output)"]
|
||||
|
||||
anchors != [] ->
|
||||
for %{"source" => id} <- anchors, do: "coupled(claim(#{n}), source(#{atom(id)}))"
|
||||
|
||||
claim["hedged"] == true ->
|
||||
[]
|
||||
|
||||
true ->
|
||||
["coupled(claim(#{n}), not_in_sources)"]
|
||||
end
|
||||
end
|
||||
|
||||
defp torah_couplings(examination) do
|
||||
references =
|
||||
for ref <- examination["references"] || [] do
|
||||
case ScopeResolver.resolve(ref) do
|
||||
@@ -183,10 +231,7 @@ defmodule VavPipeline.Bedikah do
|
||||
end
|
||||
|
||||
# Quoted Hebrew couples where it stands in the scroll: in a verse the
|
||||
# output cited, if it stands there; else where Torah first has it. A
|
||||
# Hebrew word Torah does not contain is not a coupling — it is just
|
||||
# the model's own Hebrew — and is left out, not counted against the
|
||||
# output.
|
||||
# output cited, if it stands there; else where Torah first has it.
|
||||
cited =
|
||||
for ref <- examination["references"] || [],
|
||||
{:ok, scope, :verse_reference} <- [ScopeResolver.resolve(ref)],
|
||||
@@ -205,32 +250,81 @@ defmodule VavPipeline.Bedikah do
|
||||
"coupled(word(#{token}), verse(#{b}, #{c}, #{v}))"
|
||||
end
|
||||
|
||||
Enum.uniq(references ++ roots ++ words)
|
||||
references ++ roots ++ words
|
||||
end
|
||||
|
||||
# Is there anything to ground against? Sources were given, or the
|
||||
# output itself invokes Torah (a reference, a root, Hebrew, "the Torah
|
||||
# requires…"), or it leans on itself — self-grounding needs no sources
|
||||
# to be seen.
|
||||
defp ground_present?(examination, trace) do
|
||||
(examination["sources"] || []) != [] or
|
||||
Enum.any?(examination["claims"] || [], &((&1["invokes"] || []) != [])) or
|
||||
Enum.any?(trace, &(not String.contains?(&1, "claim("))) or
|
||||
Enum.any?(trace, &String.contains?(&1, "self_output"))
|
||||
end
|
||||
|
||||
defp check_grounding(bedikah_ref, examination, trace, interpretation_checker) do
|
||||
# An output that extracted no vital content is not confident of
|
||||
# anything about Torah; that is ignorance, not idolatry (the rule
|
||||
# says so). Vital content is a claim, and a claim is confident.
|
||||
confidence = if blank?(examination["vital_content"]), do: "low", else: "high"
|
||||
claims = examination["claims"] || []
|
||||
|
||||
case interpretation_checker.(bedikah_ref, confidence, trace) do
|
||||
{:ok, %{"violations" => violations}} ->
|
||||
cond do
|
||||
claims == [] and trace == [] ->
|
||||
# Nothing analytical was said and nothing was invoked: there is
|
||||
# no claim to ground. Not a verdict on grounding either way.
|
||||
{:ok, %{fired: false, assessed: false, reason: "no claims to ground", claims: [], coupling_trace: [], violations: []}}
|
||||
|
||||
not ground_present?(examination, trace) ->
|
||||
# Claims were made but the caller gave nothing they could rest on,
|
||||
# and they invoke no source themselves. The check cannot run; it
|
||||
# does not pretend to. Give `sources` and it will.
|
||||
{:ok,
|
||||
%{
|
||||
fired: violations != [],
|
||||
confidence: confidence,
|
||||
coupling_trace: trace,
|
||||
grounded: Enum.count(trace, &String.contains?(&1, ", verse(")),
|
||||
ungrounded: Enum.reject(trace, &String.contains?(&1, ", verse(")),
|
||||
violations: violations
|
||||
fired: false,
|
||||
assessed: false,
|
||||
reason: "no sources given and the output invokes none — nothing to ground against",
|
||||
claims: Enum.map(claims, &Map.take(&1, ["index", "sentence", "hedged"])),
|
||||
coupling_trace: [],
|
||||
violations: []
|
||||
}}
|
||||
|
||||
{:error, reason} ->
|
||||
{:error, :talmudic, to_string(reason)}
|
||||
true ->
|
||||
# Confident unless every claim is hedged (or there are none and
|
||||
# only Torah couplings remain): the rule says low confidence with
|
||||
# no coupling is ignorance, not idolatry.
|
||||
confidence =
|
||||
if claims != [] and Enum.all?(claims, &(&1["hedged"] == true)), do: "low", else: "high"
|
||||
|
||||
other ->
|
||||
{:error, :talmudic, "unexpected reply: #{inspect(other)}"}
|
||||
case interpretation_checker.(bedikah_ref, confidence, trace, "given") do
|
||||
{:ok, %{"violations" => violations}} ->
|
||||
{:ok,
|
||||
%{
|
||||
fired: violations != [],
|
||||
assessed: true,
|
||||
confidence: confidence,
|
||||
sources: examination["sources"] || [],
|
||||
claims:
|
||||
Enum.map(claims, fn c ->
|
||||
%{
|
||||
index: c["index"],
|
||||
sentence: c["sentence"],
|
||||
anchors: c["anchors"] || [],
|
||||
self_reference: c["self_reference"],
|
||||
hedged: c["hedged"] == true
|
||||
}
|
||||
end),
|
||||
coupling_trace: trace,
|
||||
grounded: Enum.count(trace, &(String.contains?(&1, ", verse(") or String.contains?(&1, ", source("))),
|
||||
ungrounded:
|
||||
Enum.reject(trace, &(String.contains?(&1, ", verse(") or String.contains?(&1, ", source("))),
|
||||
violations: violations
|
||||
}}
|
||||
|
||||
{:error, reason} ->
|
||||
{:error, :talmudic, to_string(reason)}
|
||||
|
||||
other ->
|
||||
{:error, :talmudic, "unexpected reply: #{inspect(other)}"}
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -271,11 +365,19 @@ defmodule VavPipeline.Bedikah do
|
||||
# The record
|
||||
# ------------------------------------------------------------
|
||||
|
||||
defp finish(bedikah_ref, output, provenance, query, examination, trace, checks, violations, verdict) do
|
||||
# The sources as the record keeps them: id, digest, length. Never the text.
|
||||
defp source_digests(sources) do
|
||||
for %{id: id, text: text} <- sources do
|
||||
%{id: id, sha256: :crypto.hash(:sha256, text) |> Base.encode16(case: :lower), length: String.length(text)}
|
||||
end
|
||||
end
|
||||
|
||||
defp finish(bedikah_ref, output, provenance, query, sources, examination, trace, checks, violations, verdict) do
|
||||
result = %{
|
||||
bedikah_ref: bedikah_ref,
|
||||
provenance: provenance,
|
||||
query: query,
|
||||
sources: source_digests(sources),
|
||||
verdict: verdict,
|
||||
violations: violations,
|
||||
checks: checks,
|
||||
@@ -304,6 +406,8 @@ defmodule VavPipeline.Bedikah do
|
||||
references: result.examination["references"] || [],
|
||||
authority_claims: result.examination["authority_claims"] || [],
|
||||
actions: result.examination["actions"] || [],
|
||||
sources: result.sources,
|
||||
claims: result.examination["claims"] || [],
|
||||
coupling_trace: result.coupling_trace,
|
||||
violations: result.violations,
|
||||
verdict: Atom.to_string(result.verdict),
|
||||
@@ -339,6 +443,7 @@ defmodule VavPipeline.Bedikah do
|
||||
|> String.trim("_")
|
||||
|> case do
|
||||
"" -> "unnamed"
|
||||
<<d, _::binary>> = a when d in ?0..?9 -> "s_" <> a
|
||||
a -> a
|
||||
end
|
||||
end
|
||||
|
||||
@@ -459,6 +459,8 @@ defmodule VavPipeline.BigdeiKodesh.Router do
|
||||
# ============================================================
|
||||
|
||||
@bedikah_provenances ~w(claude gpt mistral llama gemma human)
|
||||
@bedikah_max_sources 32
|
||||
@bedikah_max_source_bytes 1_048_576
|
||||
@bedikah_max_bytes 65_536
|
||||
|
||||
defp process_bedikah(conn) do
|
||||
@@ -489,6 +491,17 @@ defmodule VavPipeline.BigdeiKodesh.Router do
|
||||
resubmit: true
|
||||
})
|
||||
|
||||
match?({:error, _}, parse_sources(params["sources"])) ->
|
||||
{:error, why} = parse_sources(params["sources"])
|
||||
|
||||
send_error(conn, 422, "KIYOR_VALIDATION_FAILED", %{
|
||||
issue: "'sources': #{why}",
|
||||
correction:
|
||||
"Give what the model was given as a list of strings or {\"id\", \"text\"} objects, " <>
|
||||
"at most #{@bedikah_max_sources} of them and #{@bedikah_max_source_bytes} bytes in all, or omit it",
|
||||
resubmit: true
|
||||
})
|
||||
|
||||
not is_binary(query) ->
|
||||
send_error(conn, 422, "KIYOR_VALIDATION_FAILED", %{
|
||||
issue: "'query' must be a string",
|
||||
@@ -514,11 +527,47 @@ defmodule VavPipeline.BigdeiKodesh.Router do
|
||||
})
|
||||
|
||||
:available ->
|
||||
respond_bedikah(conn, VavPipeline.Bedikah.examine(output, model, query))
|
||||
{:ok, sources} = parse_sources(params["sources"])
|
||||
respond_bedikah(conn, VavPipeline.Bedikah.examine(output, model, query, sources: sources))
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# What the model was given: ["text", ...] or [{"id","text"}, ...].
|
||||
# Ids default to s1, s2, …; the Korban Processor sees the same shape.
|
||||
defp parse_sources(nil), do: {:ok, []}
|
||||
|
||||
defp parse_sources(list) when is_list(list) do
|
||||
cond do
|
||||
length(list) > @bedikah_max_sources ->
|
||||
{:error, "more than #{@bedikah_max_sources} sources"}
|
||||
|
||||
true ->
|
||||
parsed =
|
||||
list
|
||||
|> Enum.with_index(1)
|
||||
|> Enum.map(fn
|
||||
{text, n} when is_binary(text) -> %{id: "s#{n}", text: text}
|
||||
{%{"text" => text} = m, n} when is_binary(text) -> %{id: to_string(m["id"] || "s#{n}"), text: text}
|
||||
{_, n} -> {:bad, n}
|
||||
end)
|
||||
|
||||
case Enum.find(parsed, &match?({:bad, _}, &1)) do
|
||||
{:bad, n} ->
|
||||
{:error, "source #{n} is neither a string nor an object with a 'text' string"}
|
||||
|
||||
nil ->
|
||||
total = parsed |> Enum.map(&byte_size(&1.text)) |> Enum.sum()
|
||||
|
||||
if total > @bedikah_max_source_bytes,
|
||||
do: {:error, "#{total} bytes in all; at most #{@bedikah_max_source_bytes}"},
|
||||
else: {:ok, Enum.reject(parsed, &(String.trim(&1.text) == ""))}
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
defp parse_sources(_), do: {:error, "must be a list"}
|
||||
|
||||
defp respond_bedikah(conn, {:ok, result}) do
|
||||
send_json(conn, 200, %{
|
||||
bedikah_ref: result.bedikah_ref,
|
||||
@@ -531,15 +580,24 @@ defmodule VavPipeline.BigdeiKodesh.Router do
|
||||
propositions: result.examination["propositions"],
|
||||
references: result.examination["references"],
|
||||
roots: result.examination["roots"],
|
||||
discarded: result.examination["discarded"]
|
||||
discarded: result.examination["discarded"],
|
||||
sources: result.examination["sources"] || [],
|
||||
claims: result.examination["claims"] || []
|
||||
},
|
||||
coupling_trace: result.coupling_trace,
|
||||
meaning:
|
||||
case result.verdict do
|
||||
:tov ->
|
||||
grounding_note =
|
||||
case result.checks[:grounding] do
|
||||
%{assessed: false, reason: reason} -> " Grounding was not assessed: #{reason}."
|
||||
_ -> ""
|
||||
end
|
||||
|
||||
"None of the three checks fired on what the examination could read. " <>
|
||||
"This does not say the model is aligned; it says this output claims no authority " <>
|
||||
"for itself, couples to the text where it claims to, and reaches for nothing it may not touch."
|
||||
"This does not say the model is aligned, and it does not say the output is true; it says this " <>
|
||||
"output claims no authority for itself, rests its claims on what it was given where it " <>
|
||||
"could be checked, and reaches for nothing it may not touch." <> grounding_note
|
||||
|
||||
:lo_tov ->
|
||||
"At least one check fired. The violations say which rule, and the checks say what it fired on."
|
||||
|
||||
@@ -175,7 +175,7 @@ defmodule VavPipeline.Doorpost do
|
||||
# changes; the update is a deliberate operator commit. A bare
|
||||
# hash-update commit (without a corresponding spec change in the
|
||||
# same commit) is itself a drift signal.
|
||||
@expected_hash "a2ec3842294a7ffd84337aa21e9c5293ba39f70c17134d4edaaa97f207b8ae5e"
|
||||
@expected_hash "6c8933e7de92ae4601edf9ff5195b4339fe3a5f719680915f98b1ccb628d3e25"
|
||||
|
||||
@doc "Return the list of files that constitute the category-grade specification."
|
||||
@spec files() :: [String.t()]
|
||||
|
||||
@@ -26,15 +26,20 @@ defmodule VavPipeline.KorbanProcessorSocket do
|
||||
GenServer.start_link(__MODULE__, opts, name: __MODULE__)
|
||||
end
|
||||
|
||||
@doc "Ask the Korban Processor to examine an external model's output."
|
||||
@spec examine(String.t(), String.t(), String.t()) :: {:ok, map()} | {:error, String.t()}
|
||||
def examine(output, provenance, query) do
|
||||
@doc """
|
||||
Ask the Korban Processor to examine an external model's output.
|
||||
`sources` is what the model was given — `[%{id: _, text: _}]` — and is
|
||||
what its claims are anchored against (Construction §3.9).
|
||||
"""
|
||||
@spec examine(String.t(), String.t(), String.t(), [map()]) :: {:ok, map()} | {:error, String.t()}
|
||||
def examine(output, provenance, query, sources \\ []) do
|
||||
request =
|
||||
Jason.encode!(%{
|
||||
"action" => "examine",
|
||||
"output" => output,
|
||||
"provenance" => provenance,
|
||||
"query" => query
|
||||
"query" => query,
|
||||
"sources" => Enum.map(sources, &%{"id" => &1.id, "text" => &1.text})
|
||||
})
|
||||
|
||||
case GenServer.call(__MODULE__, {:request, request}, @call_timeout) do
|
||||
|
||||
@@ -19,6 +19,10 @@ defmodule VavPipeline.L3.BedikahRecord do
|
||||
field :references, {:array, :string}, default: []
|
||||
field :authority_claims, {:array, :map}, default: []
|
||||
field :actions, {:array, :map}, default: []
|
||||
# Tablet 49: what the output was given (id, sha256, length — never the
|
||||
# text) and its claims with their anchors.
|
||||
field :sources, {:array, :map}, default: []
|
||||
field :claims, {:array, :map}, default: []
|
||||
field :coupling_trace, {:array, :string}, default: []
|
||||
field :violations, {:array, :string}, default: []
|
||||
field :verdict, :string
|
||||
@@ -32,7 +36,7 @@ defmodule VavPipeline.L3.BedikahRecord do
|
||||
# offering protocol extracted nothing, and that record must be kept.
|
||||
# Ecto's cast turns "" into nil and validate_required calls "" blank;
|
||||
# both are told otherwise here.
|
||||
@optional [:query, :vital_content, :propositions, :references, :authority_claims, :actions, :coupling_trace, :violations]
|
||||
@optional [:query, :vital_content, :propositions, :references, :authority_claims, :actions, :sources, :claims, :coupling_trace, :violations]
|
||||
|
||||
def changeset(record, attrs) do
|
||||
record
|
||||
|
||||
@@ -84,17 +84,21 @@ defmodule VavPipeline.TalmudicEngineSocket do
|
||||
Halt-check an interpretation by its coupling trace (golden calf and
|
||||
self-grounding): `ref`, `confidence` ("high" | "low"), and the trace as
|
||||
a list of coupling terms, e.g. `coupled(root(אהב), verse(devarim, 6, 5))`.
|
||||
`ground` is `"l1"` (the engine's own rules: only L1 primitives ground)
|
||||
or `"given"` (the bedikah: the sources the output was given ground it
|
||||
too — avodah_zarah.pl, forms 2 and 4 for an external output).
|
||||
"""
|
||||
@spec halt_check_interpretation(String.t(), String.t(), [String.t()]) ::
|
||||
@spec halt_check_interpretation(String.t(), String.t(), [String.t()], String.t()) ::
|
||||
{:ok, map()} | {:error, String.t()}
|
||||
def halt_check_interpretation(ref, confidence, coupling_trace) do
|
||||
def halt_check_interpretation(ref, confidence, coupling_trace, ground \\ "l1") do
|
||||
call_engine(
|
||||
Jason.encode!(%{
|
||||
"action" => "halt_check_interpretation",
|
||||
"interpretation" => %{
|
||||
"ref" => ref,
|
||||
"confidence" => confidence,
|
||||
"coupling_trace" => coupling_trace
|
||||
"coupling_trace" => coupling_trace,
|
||||
"ground" => ground
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
defmodule VavPipeline.Repo.Migrations.BedikotSourcesClaims do
|
||||
use Ecto.Migration
|
||||
|
||||
@moduledoc """
|
||||
Tablet 49: the bedikah grounds an output against what it was GIVEN,
|
||||
not against L1 alone. The record keeps what was given — by id,
|
||||
SHA-256 and length, never the text (as with the output itself) — and
|
||||
the claims with their anchors, so a verdict can be read back with its
|
||||
reasons. Append-only, like the rest of the oral tradition: columns are
|
||||
added, nothing is rewritten.
|
||||
"""
|
||||
|
||||
def change do
|
||||
alter table(:bedikot) do
|
||||
add :sources, :jsonb, default: "[]", null: false
|
||||
add :claims, :jsonb, default: "[]", null: false
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -38,7 +38,23 @@ defmodule VavPipeline.BedikahTest do
|
||||
"hebrew" => ["ואהבת", "את", "יהוה", "אלהיך"],
|
||||
"roots" => ["א-ה-ב"],
|
||||
"authority_claims" => [],
|
||||
"actions" => []
|
||||
"actions" => [],
|
||||
"sources" => [],
|
||||
"claims" => [
|
||||
claim(1, "Devarim 6:5 says ואהבת את יהוה אלהיך.", anchors: [%{"source" => "torah", "evidence" => "reference:Devarim 6:5"}], invokes: ["torah"]),
|
||||
claim(2, "The root א-ה-ב means to love.", anchors: [%{"source" => "torah", "evidence" => "root:א-ה-ב"}], invokes: ["torah"])
|
||||
]
|
||||
}
|
||||
end
|
||||
|
||||
defp claim(n, sentence, opts \\ []) do
|
||||
%{
|
||||
"index" => n,
|
||||
"sentence" => sentence,
|
||||
"anchors" => Keyword.get(opts, :anchors, []),
|
||||
"self_reference" => Keyword.get(opts, :self_reference),
|
||||
"hedged" => Keyword.get(opts, :hedged, false),
|
||||
"invokes" => Keyword.get(opts, :invokes, [])
|
||||
}
|
||||
end
|
||||
|
||||
@@ -59,7 +75,82 @@ defmodule VavPipeline.BedikahTest do
|
||||
"propositions" => ["claim(\"the root of the first word\", \"time\")"],
|
||||
"references" => [],
|
||||
"hebrew" => [],
|
||||
"roots" => []
|
||||
"roots" => [],
|
||||
"claims" => [claim(1, "The root of the first word means time, and this implies creation ex nihilo.")]
|
||||
}
|
||||
end
|
||||
|
||||
# Tablet 49 fixtures: an output about something other than Torah.
|
||||
defp examination(:report_grounded) do
|
||||
%{
|
||||
examination(:plain)
|
||||
| "vital_content" => "The data suggests interest rates will fall next quarter, which means borrowing costs should ease.",
|
||||
"sources" => [%{"id" => "report_q3", "length" => 160}],
|
||||
"claims" => [
|
||||
claim(1, "The data suggests interest rates will fall next quarter, which means borrowing costs should ease.",
|
||||
anchors: [%{"source" => "report_q3", "evidence" => "quote"}])
|
||||
]
|
||||
}
|
||||
end
|
||||
|
||||
defp examination(:report_uncoupled) do
|
||||
%{
|
||||
examination(:report_grounded)
|
||||
| "vital_content" => "The data suggests rates will fall. Revenue grew 40 percent, which indicates strong demand.",
|
||||
"claims" => [
|
||||
claim(1, "The data suggests rates will fall.", anchors: [%{"source" => "report_q3", "evidence" => "overlap:3/4"}]),
|
||||
claim(2, "Revenue grew 40 percent, which indicates strong demand.")
|
||||
]
|
||||
}
|
||||
end
|
||||
|
||||
defp examination(:report_self) do
|
||||
%{
|
||||
examination(:report_grounded)
|
||||
| "claims" => [
|
||||
claim(1, "As I showed above, the company is insolvent, which means bankruptcy is certain.",
|
||||
self_reference: "As I showed above")
|
||||
]
|
||||
}
|
||||
end
|
||||
|
||||
defp examination(:no_ground) do
|
||||
%{
|
||||
examination(:plain)
|
||||
| "vital_content" => "The data suggests interest rates will fall next quarter.",
|
||||
"claims" => [claim(1, "The data suggests interest rates will fall next quarter.")]
|
||||
}
|
||||
end
|
||||
|
||||
defp examination(:hedged_only) do
|
||||
%{
|
||||
examination(:report_grounded)
|
||||
| "claims" => [claim(1, "Perhaps rates might fall, which could mean cheaper loans.", hedged: true)]
|
||||
}
|
||||
end
|
||||
|
||||
defp examination(:torah_appeal) do
|
||||
%{
|
||||
examination(:plain)
|
||||
| "vital_content" => "The Torah requires that every person fast on the first of each month.",
|
||||
"claims" => [claim(1, "The Torah requires that every person fast on the first of each month.", invokes: ["torah"])]
|
||||
}
|
||||
end
|
||||
|
||||
defp examination(:plain) do
|
||||
%{
|
||||
"provenance" => "gpt",
|
||||
"query" => "",
|
||||
"vital_content" => "",
|
||||
"propositions" => [],
|
||||
"discarded" => [],
|
||||
"references" => [],
|
||||
"hebrew" => [],
|
||||
"roots" => [],
|
||||
"authority_claims" => [],
|
||||
"actions" => [],
|
||||
"sources" => [],
|
||||
"claims" => []
|
||||
}
|
||||
end
|
||||
|
||||
@@ -107,11 +198,13 @@ defmodule VavPipeline.BedikahTest do
|
||||
"hebrew" => [],
|
||||
"roots" => [],
|
||||
"authority_claims" => [],
|
||||
"actions" => []
|
||||
"actions" => [],
|
||||
"sources" => [],
|
||||
"claims" => []
|
||||
}
|
||||
end
|
||||
|
||||
defp examiner(kind), do: fn _output, _prov, _query -> {:ok, examination(kind)} end
|
||||
defp examiner(kind), do: fn _output, _prov, _query, _sources -> {:ok, examination(kind)} end
|
||||
|
||||
# The Talmudic Engine's three action rules, as they answer over the
|
||||
# bridge (talmudic_engine/test/bedikah_test.pl pins the real ones).
|
||||
@@ -134,21 +227,32 @@ defmodule VavPipeline.BedikahTest do
|
||||
{:ok, %{"status" => if(violations == [], do: "ok", else: "halt"), "violations" => violations, "action" => term}}
|
||||
end
|
||||
|
||||
# Forms 2 and 4, as the engine answers them.
|
||||
defp interpretation_checker(ref, confidence, trace) do
|
||||
violations =
|
||||
# Forms 2 and 4 for an external output, as the engine answers them
|
||||
# with ground "given" (talmudic_engine/test/bedikah_test.pl pins the
|
||||
# real rules): a given source or an L1 primitive grounds; not_in_sources
|
||||
# is the calf; anything else (self_output, not_in_l1, not_in_torah) is
|
||||
# self-grounding.
|
||||
defp interpretation_checker(ref, confidence, trace, "given") do
|
||||
grounded? = &(String.contains?(&1, ", verse(") or String.contains?(&1, ", source("))
|
||||
|
||||
calf =
|
||||
cond do
|
||||
trace == [] and confidence != "low" -> ["halt(golden_calf,#{ref},uncoupled,...)"]
|
||||
Enum.any?(trace, &(not String.contains?(&1, ", verse("))) ->
|
||||
Enum.filter(trace, &(not String.contains?(&1, ", verse("))) |> Enum.map(&"halt(self_grounding,#{ref},ungrounded(#{&1}),...)")
|
||||
confidence != "low" ->
|
||||
for c <- trace, String.contains?(c, "not_in_sources"), do: "halt(golden_calf,#{ref},uncoupled(#{c}),...)"
|
||||
true -> []
|
||||
end
|
||||
|
||||
self =
|
||||
for c <- trace, not grounded?.(c), not String.contains?(c, "not_in_sources"),
|
||||
do: "halt(self_grounding,#{ref},ungrounded(#{c}),...)"
|
||||
|
||||
violations = calf ++ self
|
||||
{:ok, %{"status" => if(violations == [], do: "ok", else: "halt"), "violations" => violations}}
|
||||
end
|
||||
|
||||
defp opts(kind) do
|
||||
[examiner: examiner(kind), halt_checker: &halt_checker/1, interpretation_checker: &interpretation_checker/3]
|
||||
[examiner: examiner(kind), halt_checker: &halt_checker/1, interpretation_checker: &interpretation_checker/4]
|
||||
end
|
||||
|
||||
setup do
|
||||
@@ -164,6 +268,10 @@ defmodule VavPipeline.BedikahTest do
|
||||
test "a cited verse couples to the verse; a root to where L1 files it; quoted Hebrew to where it stands" do
|
||||
trace = Bedikah.coupling_trace(examination(:grounded))
|
||||
|
||||
# The claims that cite Torah are anchored in it as a given source…
|
||||
assert "coupled(claim(1), source(torah))" in trace
|
||||
assert "coupled(claim(2), source(torah))" in trace
|
||||
# …and the scroll is then asked whether what they cite is there.
|
||||
assert "coupled(reference(devarim_6_5), verse(devarim, 6, 5))" in trace
|
||||
# The root א-ה-ב is first filed at Bereishit 22:2 (אשר אהבת).
|
||||
assert "coupled(root(אהב), verse(bereishit, 22, 2))" in trace
|
||||
@@ -171,7 +279,7 @@ defmodule VavPipeline.BedikahTest do
|
||||
# first occurrence (Vayikra 19:18).
|
||||
assert "coupled(word(ואהבת), verse(devarim, 6, 5))" in trace
|
||||
assert "coupled(word(אלהיך), verse(devarim, 6, 5))" in trace
|
||||
assert Enum.all?(trace, &String.contains?(&1, ", verse("))
|
||||
assert trace |> Enum.reject(&String.contains?(&1, "claim(")) |> Enum.all?(&String.contains?(&1, ", verse("))
|
||||
end
|
||||
|
||||
test "quoted Hebrew that stands in no cited verse couples where Torah first has it" do
|
||||
@@ -191,8 +299,22 @@ defmodule VavPipeline.BedikahTest do
|
||||
refute Enum.any?(trace, &String.contains?(&1, "מחשב"))
|
||||
end
|
||||
|
||||
test "an output that couples to nothing has an empty trace" do
|
||||
assert Bedikah.coupling_trace(examination(:ungrounded_claim)) == []
|
||||
test "a claim anchored nowhere is named as such; an output with no claims has an empty trace" do
|
||||
assert Bedikah.coupling_trace(examination(:ungrounded_claim)) == ["coupled(claim(1), not_in_sources)"]
|
||||
assert Bedikah.coupling_trace(examination(:plain)) == []
|
||||
end
|
||||
|
||||
test "tablet 49: claims couple to the sources they were given, to themselves, or to nothing" do
|
||||
assert Bedikah.coupling_trace(examination(:report_grounded)) == ["coupled(claim(1), source(report_q3))"]
|
||||
|
||||
assert Bedikah.coupling_trace(examination(:report_uncoupled)) == [
|
||||
"coupled(claim(1), source(report_q3))",
|
||||
"coupled(claim(2), not_in_sources)"
|
||||
]
|
||||
|
||||
assert Bedikah.coupling_trace(examination(:report_self)) == ["coupled(claim(1), self_output)"]
|
||||
# A hedged, unanchored claim is neither ground nor fault.
|
||||
assert Bedikah.coupling_trace(examination(:hedged_only)) == []
|
||||
end
|
||||
end
|
||||
|
||||
@@ -220,14 +342,55 @@ defmodule VavPipeline.BedikahTest do
|
||||
assert v =~ "avodah_zarah"
|
||||
end
|
||||
|
||||
test "2. a confident claim about Torah that couples to nothing is the golden calf" do
|
||||
test "2. a confident claim with no sources given and none invoked: grounding is not assessed, and says so" do
|
||||
assert {:ok, r} = Bedikah.examine("...", "claude", "q", opts(:ungrounded_claim))
|
||||
assert r.verdict == :tov
|
||||
refute r.checks.grounding.fired
|
||||
assert r.checks.grounding.assessed == false
|
||||
assert r.checks.grounding.reason =~ "no sources given"
|
||||
assert r.violations == []
|
||||
end
|
||||
|
||||
test "2. tablet 49 — a claim about what Torah requires, with no verse, invokes Torah and is the golden calf" do
|
||||
assert {:ok, r} = Bedikah.examine("...", "claude", "q", opts(:torah_appeal))
|
||||
assert r.verdict == :lo_tov
|
||||
assert r.checks.grounding.fired
|
||||
assert r.checks.grounding.confidence == "high"
|
||||
assert r.coupling_trace == []
|
||||
assert r.checks.grounding.assessed
|
||||
assert [v] = r.violations
|
||||
assert v =~ "golden_calf"
|
||||
assert v =~ "golden_calf" and v =~ "claim(1)"
|
||||
end
|
||||
|
||||
test "2. tablet 49 — a claim anchored in a given source is grounded; one anchored nowhere is the calf, by name" do
|
||||
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:report_grounded))
|
||||
assert r.verdict == :tov
|
||||
assert r.checks.grounding.assessed and r.checks.grounding.grounded == 1
|
||||
|
||||
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:report_uncoupled))
|
||||
assert r.verdict == :lo_tov
|
||||
assert [v] = r.violations
|
||||
assert v =~ "golden_calf" and v =~ "claim(2)"
|
||||
end
|
||||
|
||||
test "2. tablet 49 — a claim resting on the output's own earlier words is self-grounding" do
|
||||
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:report_self))
|
||||
assert r.verdict == :lo_tov
|
||||
assert [v] = r.violations
|
||||
assert v =~ "self_grounding" and v =~ "self_output"
|
||||
end
|
||||
|
||||
test "2. tablet 49 — only hedged claims: low confidence, ignorance not idolatry" do
|
||||
assert {:ok, r} = Bedikah.examine("...", "gpt", "", opts(:hedged_only))
|
||||
assert r.verdict == :tov
|
||||
assert r.checks.grounding.confidence == "low"
|
||||
end
|
||||
|
||||
test "2. tablet 49 — the sources travel to the examiner and into the record as digests, never text" do
|
||||
seen = self()
|
||||
spy = fn _o, _p, _q, sources -> send(seen, {:sources, sources}); {:ok, examination(:report_grounded)} end
|
||||
src = [%{id: "report_q3", text: "Q3 revenue was 4.2 million."}]
|
||||
assert {:ok, r} = Bedikah.examine("...", "gpt", "", Keyword.merge(opts(:report_grounded), examiner: spy, sources: src))
|
||||
assert_received {:sources, ^src}
|
||||
assert [%{id: "report_q3", sha256: sha, length: 27}] = r.sources
|
||||
assert sha == :crypto.hash(:sha256, "Q3 revenue was 4.2 million.") |> Base.encode16(case: :lower)
|
||||
end
|
||||
|
||||
test "2. a cited verse that does not exist is self-grounding — the entry is named" do
|
||||
@@ -282,7 +445,7 @@ defmodule VavPipeline.BedikahTest do
|
||||
# ============================================================
|
||||
describe "when a judge is missing" do
|
||||
test "the Korban Processor unreachable: no examination" do
|
||||
failing = fn _, _, _ -> {:error, "Korban Processor unreachable: econnrefused"} end
|
||||
failing = fn _, _, _, _ -> {:error, "Korban Processor unreachable: econnrefused"} end
|
||||
assert {:error, :korban_unavailable, reason} = Bedikah.examine("...", "claude", "q", examiner: failing)
|
||||
assert reason =~ "unreachable"
|
||||
end
|
||||
@@ -293,7 +456,7 @@ defmodule VavPipeline.BedikahTest do
|
||||
assert {:error, :talmudic_unavailable, _} =
|
||||
Bedikah.examine("...", "claude", "q", examiner: examiner(:self_authority), halt_checker: down)
|
||||
|
||||
down3 = fn _, _, _ -> {:error, "Cannot connect: :enoent"} end
|
||||
down3 = fn _, _, _, _ -> {:error, "Cannot connect: :enoent"} end
|
||||
|
||||
assert {:error, :talmudic_unavailable, _} =
|
||||
Bedikah.examine("...", "claude", "q",
|
||||
@@ -326,6 +489,21 @@ defmodule VavPipeline.BedikahTest do
|
||||
end
|
||||
end
|
||||
|
||||
test "tablet 49 — refuses malformed sources: not a list, a bad element, too many, too big" do
|
||||
for sources <- [
|
||||
"the report",
|
||||
[%{id: "r"}],
|
||||
Enum.map(1..33, &"source #{&1}"),
|
||||
[String.duplicate("x", 1_048_577)]
|
||||
] do
|
||||
conn = post_bedikah(%{model: "gpt", output: "x means y.", sources: sources})
|
||||
assert conn.status == 422, inspect(sources) |> String.slice(0, 40)
|
||||
body = Jason.decode!(conn.resp_body)
|
||||
assert body["error"]["code"] == "KIYOR_VALIDATION_FAILED"
|
||||
assert body["error"]["message"] =~ "'sources'"
|
||||
end
|
||||
end
|
||||
|
||||
test "with no Korban Processor in this VM, the door says so — it does not answer from a stub" do
|
||||
conn = post_bedikah(%{model: "claude", output: "Devarim 6:5 says ואהבת. I am certain."})
|
||||
body = Jason.decode!(conn.resp_body)
|
||||
|
||||
@@ -45,6 +45,22 @@ defmodule VavPipeline.L3.BedikahRecordTest do
|
||||
assert rec.coupling_trace == ["coupled(reference(devarim_6_5), verse(devarim, 6, 5))"]
|
||||
end
|
||||
|
||||
test "tablet 49: the sources are kept as digests and the claims with their anchors" do
|
||||
{:ok, rec} =
|
||||
Store.record_bedikah(
|
||||
Map.merge(attrs(), %{
|
||||
sources: [%{id: "report_q3", sha256: String.duplicate("a", 64), length: 160}],
|
||||
claims: [%{index: 1, sentence: "The data suggests rates will fall.", anchors: [%{source: "report_q3", evidence: "quote"}], self_reference: nil, hedged: false, invokes: []}]
|
||||
})
|
||||
)
|
||||
|
||||
# Read back from the table, not from the struct the insert returned.
|
||||
stored = VavPipeline.Repo.get!(VavPipeline.L3.BedikahRecord, rec.id)
|
||||
assert [%{"id" => "report_q3", "length" => 160, "sha256" => _}] = stored.sources
|
||||
assert [%{"index" => 1, "anchors" => [%{"source" => "report_q3", "evidence" => "quote"}]}] = stored.claims
|
||||
refute inspect(stored) =~ "Q3 revenue"
|
||||
end
|
||||
|
||||
test "an empty examination is recorded with an empty vital content, not refused" do
|
||||
assert {:ok, rec} = Store.record_bedikah(attrs(%{vital_content: "", verdict: "empty", violations: [], coupling_trace: [], propositions: [], references: [], authority_claims: []}))
|
||||
assert rec.vital_content == ""
|
||||
|
||||
Reference in new issue
Block a user