# arXiv study finds residualization can decorrelate LLM eval scores without fixing them

> An arXiv preprint reports a preference reward model scored 0.507 on a terse-correct vs. commented-buggy MBPP choice, and that residualizing surface form shifts format effects by ~0.12 while correctness margins move under 0.01.

- **Topic**: Research
- **Published**: 2026-09-24T05:30:54.996Z
- **Canonical URL**: https://highsignal.sh/stories/arxiv-study-finds-residualization-can-decorrelate-llm-eval-scores-without-fixing-a3ff5f36

## Why It Matters

It is a caution for anyone who adjusts judge, reranker, or reward-model scores to strip out formatting: the paper argues adjustment alone is not a repair, since the removed component may carry real signal. The authors propose reporting an adjusted score only as an audit-time diagnostic beside its construct-alignment cost, never as a replacement for the raw score.

## Key Findings & Analysis

### Residualizing eval scores: gains are narrower than they look

A cross-listed arXiv CS.LG preprint (arXiv:2609.24194) argues that residualization — subtracting a score's predictable surface-form component — is not by itself a more valid measurement, because the removed component may carry construct-relevant signal and the method cannot tell which is which. The paper reports a public preference reward model choosing between a terse correct solution and a commented buggy solution for the same MBPP problem scored about like a coin flip (0.507). In designed interventions (unit-test labels with comment-only edits), residualization attenuated the reward model's format effects by about 0.12 on both correct and buggy code, while correct-versus-buggy margins moved by less than 0.01. In observational NLI and QA settings, the authors froze a held-out replication before scoring and re-evaluated it with labels from disjoint annotators; they say this supports only a narrower claim — better agreement with construct labels on a pre-declared slice where a surface-only predictor errs — not a repaired score. They add that full-population agreement fell in every observational setting with a reported positive slice gain, and within-question ranking fell in every such QA setting. The paper also reports that, in a controlled model, configurations just as damaging to construct alignment passed every pre-adjustment check, so no committed gate is a guarantee. Their proposed reporting protocol treats an adjusted score as an audit-time diagnostic reported alongside the construct-alignment cost it incurs, never a replacement for the raw score. These are the authors' own findings from a single preprint; independent replication is not reported in the supplied evidence.

## Primary Sources & Citations

- [When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits](https://arxiv.org/abs/2609.24194) — *ArXiv CS.LG (Machine Learning)* (Reporting)

---

[← Back to front page](https://highsignal.sh/) | [Daily Brief](https://highsignal.sh/brief) | [All stories](https://highsignal.sh/latest)