Deduplication sounds boring until you learn that a single 61-word sentence appears more than 60,000 times inside C4, and that language models trained on such corpora spit training-set text back at users with unnerving regularity. This long-form review of the Google paper "Deduplicating Training Data Makes Language Models Better" digs into how near-duplicate detection and long substring matching reshape the loss landscape, memorization behavior, and evaluation validity of modern LLMs.

Expect a careful walk through the two proposed dedup pipelines, the suffix-array tricks that make them tractable at web scale, and the empirical evidence that cleaner data yields ten-times-less memorization plus faster convergence to equal or better accuracy. For voice AI practitioners fine-tuning LLMs for dialogue, or training speech foundation models on transcript-heavy corpora, the message translates directly: leaked eval sentences and copy-paste generations start with duplicate training rows. The deep-dive is worth your afternoon if data pipeline decisions land on your desk.