# Peerify benchmark tests whether AI can check reviewers' claims against manuscripts

> An arXiv preprint introduces Peerify, a pipeline and 800-claim benchmark for checking whether peer-review comments are actually supported by the manuscript.

- **Topic**: Research
- **Published**: 2026-09-24T05:30:59.906Z
- **Canonical URL**: https://highsignal.sh/stories/peerify-benchmark-tests-whether-ai-can-check-reviewers-claims-against-manuscript-8b9c3b9d

## Why It Matters

Reviewer claims are usually checked by hand, and the authors report that off-the-shelf entailment models handle the task poorly, so retrieval-centered pipelines may matter more than raw model choice.

## Key Findings & Analysis

### What happened

- An arXiv preprint (CS.CL) describes **Peerify**, a pipeline that verifies whether peer-review claims are supported by the manuscript under review. Per the authors, it decomposes a review comment into atomic claims, retrieves relevant manuscript evidence, and judges whether each claim is supported (publisher: ArXiv CS.CL).

### Benchmark and findings

- The authors built a benchmark of **800 claims** drawn from authentic peer-review interactions at **NeurIPS 2024 and ICLR 2024**, including a **300-claim hand-labeled subset** used to audit the automated supervision (publisher: ArXiv CS.CL).

### Results and caveats

- Per the authors' own evaluation, automated labels agreed with human consensus on **90.3%** of audited claims (κ = 0.87), while off-the-shelf entailment models stayed below **0.24 macro-F1** (publisher: ArXiv CS.CL). These are the paper's reported results, not independent replication.

### Why it matters

- The authors report retrieval-centered verification and claim decomposition were the important factors, and that ambiguous or interpretive reviewer claims remained the hard case (publisher: ArXiv CS.CL). That points to a practical limit: automating review-claim checking may succeed more through evidence retrieval than through stronger standalone models.

## Primary Sources & Citations

- [Peerify: Benchmarking Peer-Review Claim Verification](https://arxiv.org/abs/2609.25046) — *ArXiv CS.CL (Computation and Language)* (Reporting)

---

[← Back to front page](https://highsignal.sh/) | [Daily Brief](https://highsignal.sh/brief) | [All stories](https://highsignal.sh/latest)