Behavioral change · release assurance

ProofDiff

A change should trigger the evidence its blast radius actually requires.

ProofDiff treats the candidate change—not a generic benchmark—as the starting point. It classifies changes across models, prompts, tools, MCP, policy, retrieval, and runtime configuration, then asks which behavioral contracts must be re-proven before release.

Public reference: 8 deterministic scenarios
Declared graph: 9 contracts · 2 critical
Critical path: deterministic · no LLM judge
Decision: PASS · REVIEW · BLOCK

CHANGE-CONDITIONED RELEASE

The change determines the evidence surface.

Unknown impact is treated as a reason to expand evaluation, never as permission to test less.

01
Change
model · prompt · provider · tool · MCP · policy · retrieval · runtime
02
Impact
conservative behavioral blast radius
03
Contracts
protected behaviors implicated by the change
04
Replay
deterministic fixtures + trace invariants
05
Evidence
closed bundle · provenance · comparisons · digests
06
Decision
scoped PASS · REVIEW · BLOCK

INPUT

Candidate change, not a generic benchmark run.

CONSERVATIVE RULE

Unknown high-impact changes expand the protected surface.

OUTPUT

A scoped PASS / REVIEW / BLOCK decision with its evidence boundary.

Proof moment

One refund-tool change should not force a full-graph replay.

Changed surface

tool:refund.create

Selected

4 / 9

Replay

fixed

Verdict

PASS

The pinned public reference selects four contracts and leaves five unrelated contracts skipped.

c_tool_refundc_policy_financialc_model_safetyc_tool_refund_schema

Decision semantics

Uncertainty widens. Regression still wins.

refund_tool_only

tool:refund.create

Selection4 / 9 contracts

Replayfixed

VerdictPASS

Impact-matched contracts plus the two critical contracts run.

unknown_surface

runtime:mystery_flag + tool:refund.create

Selection9 / 9 contracts

Replayunchanged

VerdictREVIEW

Unknown impact widens selection instead of silently skipping evidence.

unknown_with_regression

runtime:mystery_flag

Selection9 / 9 contracts

Replayregressed

VerdictBLOCK

Observed regression dominates novelty and review pressure.

Known boundary

A selector cannot reason over a dependency that does not exist in its declared model.

false_negative_undeclared_coupling

tool:search

Search changes can affect refund behavior in reality even when the declared graph has no search → refund edge. The selector skips c_tool_refund.

The scenario records pass: true because the expected miss was reproduced. That is a limitation record, not evidence that the regression was caught.

Evidence provenance

Public reproduction and private-core observations remain different evidence classes.

Public reference · Class A

Reproducible from the pinned checkout

Eight deterministic selection / replay / verdict scenarios.
Declared 9-contract graph with two critical contracts.
No LLM judge or network I/O on the critical decision path.
Documented false-positive and false-negative boundaries remain visible.

Historical / private-core · Class C

Not reproduced by this public repository

One preregistered CrewAI pilot; not customer validation or general recall.
96.52% private synthetic suite reduction; declared-coverage recall by construction.
Private-core evidence may inform the portfolio only when its boundary stays explicit.

Measurement contract

What would make the preferred answer fail?

EVALUATION METHOD

A candidate should ship only when the behaviors implicated by its actual change have the evidence required by release policy.

Semantic change classification, conservative impact expansion, protected-contract selection, deterministic fixture replay, trace comparison, and closed evidence bundles.

Protected omissions

implicated protected behavior not selected

Selection reduction

evidence avoided without losing required coverage

Critical regression

release-blocking contract failures detected

Unknown impact

uncertainty expands rather than shrinks evaluation

Evidence completeness

changes · contracts · traces · provenance · decision

Decision reproducibility

same evidence produces the same scoped verdict

Validation

What the evidence actually supports.

Claims stay scoped

Public reference · reproducible scenarios

VERIFIED

8 deterministic scenarios on a 9-contract declared graph

Class A public evidence. Includes a predicted false-negative miss, conservative over-selection, uncertainty, fail-closed widening, and BLOCK dominating novelty.

Private-core synthetic benchmark

SUPPORTED

300 scenarios · 2,000 contracts · 200 tools · 100% declared-coverage recall

Private-core historical artifact. Declared-coverage recall is by construction; not a production-recall or public-reference performance claim.

Private-core external validation · initial batch

SUPPORTED

Zero observed protected omissions; critical LangGraph reverse regression blocked

Initial batch only. Private-core evidence; broader external validity is not claimed and the public reference does not reproduce this result.

Decision evidence

SUPPORTED

Closed evidence bundle links changes, selected contracts, fixture traces, comparisons, policy, provenance, and decision

The decision remains scoped to declared contracts and the evidence actually available.

Engineering decisions

The implementation choices that constrain the claim.

The change determines the required evidence

A generic full-suite rerun wastes evaluation work and still does not explain why a particular behavior needed to be re-proven.

Unknown impact expands the suite

Uncertainty about blast radius must never silently reduce the set of protected behaviors evaluated.

Critical gates stay deterministic

Tool use, trajectories, state invariants, and other exact release requirements should not depend on an LLM judge.

The decision ships with its evidence boundary

PASS / REVIEW / BLOCK is only meaningful when the selected contracts, traces, comparisons, policy, provenance, and unproven areas travel with it.

Reproduction

pin 494e8cd

Public reference quickstart.py results.json claims ledger

git clone https://github.com/TensorScholar/proofdiff-reference
cd proofdiff-reference
git checkout 494e8cd448a4f5f4f13f8b84b546950b75e7f82c
make verify

all_pass: true means every documented expectation reproduced — including the predicted false-negative miss. It is not a claim of undeclared semantic completeness.

← Selected systems Next · PermitDiff