# arXiv paper: ask coding agents if a stored claim still holds, not if a diff preserves behavior

> An arXiv preprint reports that judging a specific stored claim beats judging behavior-preserving diffs when invalidation precision rises from 0.29-0.33 to 0.71-0.97.

- **Topic**: Research
- **Published**: 2026-09-24T05:35:09.940Z
- **Canonical URL**: https://highsignal.sh/stories/arxiv-paper-ask-coding-agents-if-a-stored-claim-still-holds-not-if-a-diff-preser-7beea10b

## Why It Matters

Agent memory systems that discard stored claims whenever their source artifact changes may be throwing away reliable knowledge; the paper's evidence suggests reframing the question, not upgrading the model, closes most of the accuracy gap.

## Key Findings & Analysis

### Question framing dominates model choice

A new arXiv preprint, Impact Is Not Invalidation: Ask About the Claim, Not the Diff, argues that coding-agent memory systems should test whether a specific stored claim still holds, rather than whether a commit preserves behavior. The authors report that content anchoring — invalidating a claim whenever the artifact it came from changes — fires constantly, and that semantic-equivalence classification asks about the diff instead of any stored claim. Asked whether a commit preserves behavior, five models spanning a 40x price range flagged 59-72% of real commits and reached precision of only 0.291 to 0.329 against a 0.25 base rate. Asked instead whether one specific claim still holds, the same models on the same diffs reached 0.705 to 0.974. A control that gave the behavior-preservation judge the claim text, changing only the question, moved precision by 0.010 and 0.016, while changing the question moved it by 0.49 and 0.65 — the authors' evidence that the gap is about the question, not model capability.

### A deployed selector doesn't close the gap

The paper compares against pytest-testmon, a deployed regression-test selector using coverage-derived dependency data. It reached 0.868 recall at 0.415 precision, which the authors say shows near-complete knowledge of what a change can touch does not identify what it falsifies.

### Ground truth and dataset

The authors define a claim as a test function passing at commit t, and say it has flipped if the same assertion text fails at t+1 — execution, not annotation, is the ground truth. They note a construction problem: on a CI-gated mainline, a commit that leaves a pre-existing test failing cannot merge, so the naive positive class is empty by design. They report 10,369 claims with 184 execution-verified flips mined from 23 Python libraries, with splits held out by repository, a post-knowledge-cutoff split, a shuffled-diff null, a paraphrase control, and a leave-one-repository-out analysis over 17 repositories.

## Primary Sources & Citations

- [Impact Is Not Invalidation: Ask About the Claim, Not the Diff](https://arxiv.org/abs/2609.25130) — *ArXiv CS.CL (Computation and Language)* (Reporting)
- [Impact Is Not Invalidation: Ask About the Claim, Not the Diff](https://arxiv.org/abs/2609.25130) — *ArXiv CS.AI* (Reporting)

---

[← Back to front page](https://highsignal.sh/) | [Daily Brief](https://highsignal.sh/brief) | [All stories](https://highsignal.sh/latest)