ArXiv paper cuts Transformer sequence length 40% with span-collapsing merge module

The claim

> - A preprint on arXiv (arXiv:2609.27233, ArXiv CS.CL) introduces a method for distilling sequential computation in Transformer language models by replacing spans of input tokens with collapsed representations, computed on the fly by a lightweight merge module, the authors report. The module generates a single surrogate embedding from a sequence of static token embeddings meant to capture the functional role of the multiple tokens, letting pretrained models run on compressed inputs without architectural changes or re-training. > - The authors describe applying the approach at inference to compress both prompts and intermediate decoding steps, using a rollback mechanism that substitutes stored multi-token KV cache entries with their single-step surrogates. > - Reported results across diverse models: up to 40% reduction in effective sequence length with minimal accuracy degradation on language modeling evaluations and downstream tasks, including question answering, summarization, commonsense reasoning, and long-form mathematical reasoning, per the abstract. The authors also report that additional lightweight adaptation of the merge module further improves the accuracy-compression trade-off in selected settings. > - Why it matters: the finding is an author claim from a v1 preprint (dated 2026-09-24 on arXiv) and has not been independently verified. If it holds, it points to a way to approximate, rather than recompute, sequential token computation at inference — a lever on the growing cost of long contexts. > - Caveat: the abstract describes results as approximating original model behavior without model updating; no benchmark tables, baselines, or third-party evaluations are included in the supplied evidence, and the paper is listed with no social engagement recorded. [1]