Test pertinence certificates

An experimental Lean 4 check of whether a concrete value oracle follows from the formal leaf claim for which the test was generated.

Experimental add-on · implementation at commit 42cde4c

Overview

Let $C$ be a formal leaf claim about an opaque program $P$. A test expects output $y$ at input $x$. For the supported fragment, the test is called pertinent to $C$ when Lean accepts a proof of:

$$C \;\Longrightarrow\; P(x)=y$$

Because $P$ is opaque, the proof cannot inspect an implementation. It establishes that the test oracle is a logical consequence of the formal claim, not that a particular implementation passes the test.

A certified case is removed from the test-applicability Human Obligation. Uncertified, unsupported, and operationally failed cases stay in its scope. The independent suite-adequacy obligation is always retained and remains non-blocking.

Thus, if a suite contains $n$ cases and $k$ are certified, human applicability review is narrowed to $n-k$ cases. The certificate says nothing about whether those $k$ cases are collectively adequate.

Current arity: certification supports one program symbol with exactly one positional argument. This is a restriction of the bridge and the current PROG : INPUT → OUTPUT interface, not of Lean. A unary input may be a supported list or option, but products and structures—and therefore general argument packing—are not yet encoded.

Implementation · theorem construction and obligation rewrite

theorem pertinent_t : Claim → PROG input = expected := by
  ...

certified                 → omit this case from applicability review
uncertified | unsupported → retain this case
error                     → retain this case
all outcomes              → retain suite-adequacy review

Certification service · Human Obligations

1. The checked judgment

The formalization environment declares two types and an uninterpreted program constant:

def INPUT  : Type := ...
def OUTPUT : Type := ...
opaque PROG : INPUT → OUTPUT

A leaf claim is a proposition $C : \mathsf{Prop}$ in that environment. For a generated test case with a single positional input $x$ and a value oracle $y$, the bridge constructs closed Lean terms $\ulcorner x\urcorner : \mathsf{INPUT}$ and $\ulcorner y\urcorner : \mathsf{OUTPUT}$. The certification obligation is:

$$\Gamma \vdash C \to \mathsf{PROG}\;\ulcorner x\urcorner = \ulcorner y\urcorner$$

Here $\Gamma$ is the imported claim module and its dependencies. “Certified” means that Lean accepted this theorem and the subsequent axiom audit found no dependency outside the configured allowlist. Pertinence is therefore relative to the formal claim, the encoder, and the imported Lean environment.

Why the implication has this direction

The intended question is whether the claim justifies the oracle. The converse, $P(x)=y \to C$, would ask whether one observation establishes an entire claim and is almost always too strong. The chosen direction also exposes underspecification: if $C$ allows several outputs at $x$, no particular equality need be derivable even when the proposed test is compatible with some model of $C$.

Failure to prove the implication is not a proof of its negation. The proof search is deliberately incomplete, and an underdetermined claim may admit the oracle without entailing it.

Implementation · exact theorem skeleton

def theorem_skeleton(name, proposition, input_term, output_term):
    return f"theorem {name} :\n    {proposition} → " \
           f"PROG {input_term} = {output_term} := by sorry"

The placeholder occurs only in the system-constructed skeleton. It is replaced before checking; candidate proofs containing sorry are rejected.
Proof checker

2. Interpreting the result

The result type separates logical success, search incompleteness, encoding limits, and infrastructure failure. Only one state changes the scope of human review.

Status	Meaning	Applicability obligation
certified	Lean accepted the exact entailment and the axiom audit passed.	Remove this case.
uncertified	Available proof attempts did not establish the entailment.	Retain this case.
unsupported	The case lies outside the declared bridge fragment.	Retain this case.
error	Formal files, the model, or Lean were unavailable, or the audit was inconclusive.	Retain this case.

A run is marked partial when at least one case has status error. Uncertified and unsupported cases do not make a run partial: they are ordinary, explicit outcomes of a completed pass.

Vacuity

If $C$ is inconsistent, every oracle follows from it. This experiment does not establish satisfiability or realizability of a leaf claim before certifying its tests. A certificate is therefore an entailment certificate, not evidence that the antecedent has a model. Non-vacuity requires a separate argument, for example a model of $C$ or a witness program satisfying it.

Implementation · result construction

proved                 → status = "certified"
proof attempts fail    → status = "uncertified"
UnsupportedCase        → status = "unsupported"
tool/model/audit fault → status = "error"

run.status = "partial" if summary["error"] else "completed"

Persisted result model

3. The JSON–Lean bridge

A theorem about the wrong representation would be irrelevant even if its proof were valid. The implementation therefore uses a small partial encoder rather than guessing a correspondence for arbitrary Python and Lean values.

Accepted formal type	Accepted test value	Lean term
`Bool`	JSON boolean	`true` or `false`
`String`	JSON string	escaped string literal
`Nat`	non-negative JSON integer	natural-number literal
`Int`	JSON integer	integer literal
`List α`	JSON array, recursively encodable	list literal
`Option α`	`null` or recursively encodable value	`none` or `some t`

Certification currently requires exactly one contract symbol, a matching test target, exactly one positional argument, no keyword arguments, and a value oracle. It cannot certify a call such as f(x, y). A function whose public interface already accepts one list or option value remains unary and may be supported; that is distinct from encoding several positional arguments as a tuple. Float, product, structure, enum, exception, and smoke-test encodings are refused. Lists and options may be nested over the supported fragment.

This bridge belongs to the trusted computing base of the claim “the generated test is represented by the checked equality.” Lean checks that the emitted terms type-check; it does not establish that the JSON-to-Lean interpretation matches an external runtime ABI. Extending the fragment requires an explicit representation argument, not merely a serializer that happens to emit accepted syntax.

Implementation · partial encoder

parse_supported_type : Lean syntax ⇀ codec description
render_value         : JSON value × codec ⇀ Lean term
render_case          : TestCase × Contract × vocabulary ⇀ (input, output)

None      ↦ none
value     ↦ some (render_value value)
[x₁,…,xₙ] ↦ [render_value x₁, …, render_value xₙ]

Bridge implementation

4. Proof production and checking

Proof discovery and proof acceptance are separate. A fixed tactic portfolio is tried first. If it fails normally, an LLM receives the vocabulary, the formal claim, the exact theorem, and Lean's previous diagnostic. It may return a tactic proof or decline. At most two LLM candidates are attempted.

The system constructs the theorem statement; the model cannot choose it.
A candidate must begin with by.
Candidates containing declaration commands, proof holes, native_decide, or comment delimiters are rejected before Lean runs.
The candidate replaces the skeleton's placeholder in a fresh Lean file.
Lean checks that exact source against the leaf-claim module.
#print axioms reports the accepted theorem's dependencies.
The certificate is issued only if the report is present and all dependencies are allowed.

The current allowlist is propext, Classical.choice, and Quot.sound. A missing or unparsable axiom report withholds the certificate. Each candidate is checked in its own file, so a rejection or tool failure is local to one case.

The lexical filter is defense in depth, not the soundness argument. The relevant authority is successful elaboration and kernel checking followed by the axiom audit. The trusted base still includes Lean, the imported modules, the checker invocation, the axiom-report parser, and the bridge described above.

Implementation · deterministic tactics and checked source

by
  intro h
  first
    | exact h input
    | exact h input (by decide)
    | exact h input (by simp)
    | simpa [Claim] using h input
    | simpa [Claim] using h input (by decide)
    | simp_all [Claim]

#print axioms pertinent_...

Proof checking and audit · Proof-search prompt

5. Obligation transformation

Applicability and adequacy are represented as separate Human Obligations beneath each test-suite tool obligation. Let $I_o$ be the case indices in suite obligation $o$, and let $D_o \subseteq I_o$ be the indices with status certified in the current run. The rewritten applicability scope is:

$$U_o = I_o \setminus D_o$$

An applicability obligation is present exactly when $U_o$ is non-empty, and its text names those indices. Existing obligation identifiers are preserved when possible. Other child obligations are left untouched. The adequacy obligation is reconstructed unconditionally, including migration of the earlier combined review obligation.

For ten cases, ten certificates remove the applicability obligation; seven certificates leave an obligation scoped to the remaining three. In both cases the adequacy obligation remains. Neither obligation blocks rationale development, and no test is rejected or removed from its suite by certification.

Implementation · scope rewrite

unresolved[obligation_id] = {
    c.case_index for c in certificates
    if c.status != "certified"
}

children = preserved_children + [adequacy_obligation(...)]
if unresolved_indices:
    children.append(applicability_obligation(..., unresolved_cases))

Obligation rewrite

6. Identity, freshness, and records

Certification is an explicit operation over the current test-plan snapshot. The run fingerprint combines the specification, interface contract, leaf identifiers and descriptions, all planned cases, the codec version, and SHA-256 hashes of every leaf's formal source and vocabulary source.

Each case also has a subject hash binding its claim and obligation identifiers, case index and payload, claim and vocabulary source hashes, codec version, and exact theorem. These hashes are content identifiers, not signatures or attestations by an external authority.

A completed run with the same fingerprint is reused unless retry=True. A changed snapshot is reported as stale and triggers a new run on the next certification call. Partial runs are not reused. Since the fingerprint is computed from claims and cases rather than Human Obligation text, narrowing an obligation does not immediately invalidate the run that justified it.

The session-level bucket stores the full history, including the exact theorem, accepted proof, and complete checked Lean source for certified cases. It sits outside the rationale tree so failed or exploratory attempts do not become rationale claims.

Implementation · fingerprints, freshness, and persistence

run fingerprint = H(test-plan fingerprint,
                    codec version,
                    H(formal source) for each leaf,
                    H(vocabulary source) for each leaf)

artifacts/test_certification/bucket.json
  runs[]
    certificates[]
      subject, theorem, proof, proved_by, lean_source, status

Record types · Engine API and freshness · Bucket persistence

7. Relation to alignment review

Certification is claim-relative and does not replace the post-generation alignment review. The review pass looks for cases that do not match their natural-language claim and for contradictions across suites; its findings are placed in a separate inspection bucket. Certification instead asks a narrower formal question for each retained case.

In particular, certificates do not establish joint consistency of leaf claims. If $C_1$ entails $P(x)=y_1$ and $C_2$ entails $P(x)=y_2$, where $y_1 \ne y_2$, both cases may be individually certified. The pair then witnesses tension between the claims rather than a defect in either certificate. Detecting or proving absence of that situation requires reasoning over the conjunction of claims, which this experiment does not perform.

Implementation · separate review and certification entry points

review_test_plan(...)    → advisory findings and quarantine bucket
certify_test_plan(...)   → Lean entailment records and obligation narrowing

get_test_review_bucket(...)
get_test_certification(...)

Alignment review · Engine entry points

8. Limits and open proof obligations

The current certificate supports one precise conclusion: relative to the imported Lean environment and accepted axioms, the formal leaf claim entails the encoded test equality. It does not establish any of the following:

that the natural-language claim was formalized correctly;
that the formal claim is satisfiable or realizable by a program;
that the runtime function and values implement the Lean-level PROG, INPUT, and OUTPUT interpretation;
that the test suite is adequate, complete, minimal, or effective at finding faults;
that the leaf claims or their certified tests are jointly consistent;
that an uncertified test contradicts its claim; or
that a proof will replay under a different Lean, Mathlib, or imported-module environment.

Two current engineering limits are especially relevant to auditability. First, the run fingerprint does not include the Lean or Mathlib toolchain version. Second, the bridge is validated by code and tests rather than by a proved abstraction relation connecting JSON values, runtime calls, and Lean terms. The stored checked source makes independent replay possible, but replay is not yet performed automatically in a pinned hermetic environment.

Natural extensions of the argument

Bind the Lean toolchain, dependency lock, and imported-module closure into the certificate identity.
State and prove representation relations for each supported runtime/Lean type pair.
Pair pertinence with a non-vacuity check such as $\exists P, C(P)$.
Check joint consistency or compatible realizability of the active leaf claims.
Replay stored checked sources in an isolated, pinned checker before treating records as portable certificates.
Extend arity and data types only together with explicit encoding semantics.

Implementation · tested safety properties

• ambiguous call and oracle shapes are refused
• proof holes, declaration injection, and comments are rejected
• a missing axiom report fails closed
• unsupported cases remain human work
• partial discharge never removes adequacy review
• formal or vocabulary source changes make a run stale

Certification tests

This page describes the experimental implementation on the stacked branch experiment/test-pertinence-certificates. It is an account of the current mechanism, not a claim that the open proof obligations above have been discharged.