Near-Deduplication Strategies for Large Text Corpora

Near-duplicate text is a structural feature of how the modern web gets built and scraped, not a minor annoyance sitting at the margins of large corpora. It's a structural feature of how the modern web gets built and scraped, and it touches four things practitioners actually care about: data quality, how much compute a model needs to hit a given accuracy, how exposed a dataset owner is to copyright claims, and whether benchmark scores mean what they say they mean. CommonCrawl, the roughly 300-billion-page dataset that underpins a large share of modern pretraining, is somewhere between 14% and 52% near-duplicate records, depending on the method used to count. That range alone should end any assumption that raw crawl data is mostly unique text waiting to be cleaned up around the edges.

The efficiency cost is not abstract. Deduplicating training data cuts the rate at which models emit memorized text by a factor of about ten, and gets models to the same accuracy in fewer training steps. There's also a scaling problem: heavy redundancy doesn't just waste compute, it bends the scaling curve itself, so that adding more (duplicated) data produces gains below what the scaling laws predict for genuinely new data. Redundancy, in other words, isn't neutral noise sitting on top of a clean signal. It actively distorts the relationship between how much data goes in and how good the model comes out.

How duplication manifests at different levels of a corpus

Redundancy occurs at three distinct levels, and conflating them is where a lot of deduplication pipelines go wrong.

Document-level duplication is the obvious case: two pages that are, for practical purposes, the same document. Scraped mirrors, syndicated news articles picked up by a dozen outlets, press releases republished verbatim. Subdocument-level duplication is subtler. A cooking blog and a legal filing might share almost nothing in substance, but both may carry the same publishing platform's navigation bar, the same cookie notice, the same copyright boilerplate at the footer. The documents are distinct; the templates around them are not. Semantic-level duplication is the hardest to see and the easiest to miss entirely, appearing as a paraphrase, a machine translation, or a reworded summary of the same underlying claim. None of these share surface tokens in any way a string-matching tool would notice, yet they carry the same information twice.

The subdocument case deserves particular attention because the failure mode is not a tuning mistake, it's baked into the method. Lowering the similarity threshold on document-level comparison makes the pipeline start catching shared templates, which is good. But push that threshold low enough and it starts deleting documents that share only a boilerplate footer alongside pages of genuinely distinct content. Raising the threshold to protect those documents stops the pipeline from catching boilerplate. There is no setting that solves both problems at once, because the tradeoff is a property of treating the whole document as one similarity unit, not a flaw in any particular threshold choice.

The design decision underlying everything that follows is the unit of deduplication. Whole documents, paragraphs, arbitrary substrings, or an embedding representing meaning rather than surface form? The answer depends on the corpus and the goal, and the rest of this piece works through three families of method built around different answers: lexical and hashing-based approaches, substring-based approaches built on suffix arrays, and semantic approaches built on embeddings.

MinHash LSH: how the dominant lexical method works and where it breaks

MinHash locality-sensitive hashing (LSH) is the workhorse of large-scale lexical deduplication, and its limitations follow directly from how it's built, so the mechanism deserves a detailed look.

Each document gets represented as a set of overlapping shingles, typically n-grams of characters or words. MinHash then compresses that set into a signature of length p: a series of hash values where agreement between two documents' signatures, position by position, gives an unbiased estimate of Jaccard similarity between the underlying shingle sets. That signature is far smaller than the original document, which is the whole point.

But even with compressed signatures, comparing every document against every other document is a quadratic operation, and quadratic cost is not something that survives contact with a corpus of hundreds of millions or billions of documents. LSH solves that by splitting the signature into B bands of R rows each, hashing each band into a 64-bit key, and flagging two documents as candidate duplicates only if they share at least one band key. Instead of pairwise comparison across the whole corpus, documents only get compared within the buckets their band keys land in.

Getting these hyperparameters wrong reduces deduplication accuracy and drives up compute costs. RefinedWeb's published configuration uses 5-grams, 450 buckets of 20 minhashes each (9,000 minhashes total), hashed with sha1. Other pipelines use leaner setups, such as 256 hashes total with a Jaccard threshold of 0.85. The choice of B and R is the actual lever being pulled: more bands means a document needs to match on fewer rows to trigger a collision, which raises recall but lowers precision, pulling in more false positives along with the true duplicates. Fewer bands does the reverse. There's no universally correct setting; it's a decision about which kind of error a given project can tolerate more of.

What MinHash LSH cannot do is just as important as what it can. A single sentence inserted or deleted midway through a document shifts every subsequent shingle boundary, which can degrade the similarity signal even between documents that are otherwise identical. Heavy paraphrasing or reformatting has the same effect. And, tying back to the earlier point, subdocument boilerplate embedded in otherwise-unique documents often doesn't move the Jaccard estimate enough to trigger a collision at all, so it survives MinHash filtering untouched.

Exact hashing and suffix arrays: what approximate methods miss

MinHash is approximate by design. There's a category of duplicates that only exact, byte-level comparison will find, and a separate category that only substring-level analysis will find. Neither approximation is a strict subset of the other.

On a set of 13,197 user prompts from WildChat, byte-exact deduplication catches 5.81% of duplicates, while MinHash LSH catches 31.32%. The gap in raw numbers is notable on its own, but the more important fact is that the two methods largely catch different duplicates rather than one being a superset of the other. That's a strong argument for running both rather than treating MinHash as a complete replacement for exact matching.

Suffix arrays go after a different unit entirely: the substring. The method concatenates every document in the corpus into one long sequence, builds a suffix array over it in linear time, and uses that structure to identify duplicated substrings, also in roughly linear time. Lee et al.'s deduplication pipeline, for instance, removes every duplicated substring longer than 50 BPE tokens, a threshold that itself is a judgment call about how aggressive the removal should be. What this catches that MinHash structurally cannot: a long block of boilerplate, quoted legal text, or a repeated code snippet embedded inside two large, otherwise-distinct documents. That block might be verbatim repeated text, but if it's a small fraction of either document's total content, it won't shift the Jaccard similarity enough to register as a MinHash collision.

The two methods pair well in sequence, and the RefinedWeb pipeline demonstrates why: running exact substring deduplication after a MinHash pass cuts the size of the dataset the substring tool has to process by nearly 40%, since MinHash has already thinned out the large, easy document-level duplicates. MinHash acts as a cheap pre-filter that makes the more expensive suffix-array pass affordable at scale.

A 2025 comparison from Fraunhofer and Lamarr evaluated five deduplication algorithms side by side: MinHash/LSH, exact hashing, SimHash, scalable Bloom filters, and suffix arrays. Suffix arrays showed distinct precision and recall behavior compared to the other four methods, with tradeoffs that differ meaningfully from the approximate approaches. But that interpretability comes at a cost: it's also computationally expensive to run at real corpus scale, so it tends to appear as a second-pass tool rather than a first-pass one.

Subdocument deduplication: targeting repeated regions without discarding the document around them

Document-level deduplication, no matter how it's tuned, applies one similarity threshold to the entire document. That's the core limitation, and it cuts in both directions: set the threshold to catch shared templates and it over-removes documents that only share a template, set it to protect those documents and it under-removes cases where a long boilerplate block is a small fraction of an otherwise-unique document's total content.

Tencent's Hunyuan team built a pipeline around treating this as a segmentation problem rather than a thresholding problem. Documents get split at natural boundaries: paragraphs, sentences, lines, rather than being treated as one atomic unit for comparison purposes. Each segment then gets counted using normalized exact hashing, aggregated across the entire corpus rather than shard by shard, which closes a real gap in suffix-array pipelines: they typically only see duplicates within a single processing shard and miss cross-shard repetition.

The most distinctive part of the design is the retention policy: rather than a fixed rule for how many copies of a repeated segment to keep, the system uses an adaptive copy budget based on how frequently a segment appears and how long it is. A boilerplate line that appears across millions of documents gets deleted almost everywhere it appears. A quoted passage that appears twice gets treated far more leniently. Prior hash-based dedup systems generally apply one fixed retention rule corpus-wide, which can't accommodate a distribution where boilerplate frequency and quotation frequency differ by many orders of magnitude within the same dataset.

The benchmark numbers back the approach up. Across multiple benchmarks, the version of the pipeline that adds document-level MinHash on top of subdocument dedup averages 52.90, with the best individual scores on four of the eight (TriviaQA, HellaSwag, PIQA, MATH). The version without document-level dedup averages 52.92, essentially the same. Both configurations beat FineWeb-Edu by just over a point (1.01 and 1.03 respectively). The near-identical scores across the two configurations suggest the subdocument layer is doing most of the real work, and document-level MinHash is adding little on top of it once segmentation-based dedup is already in place.

Semantic deduplication: catching meaning-level redundancy that lexical methods cannot reach

Every method covered so far operates on surface tokens: shingles, byte sequences, substrings. That's a hard ceiling. Two documents that say the same thing in different words share no meaningful n-gram overlap and will sail straight through MinHash without a single collision. Semantic duplication is, by construction, invisible to lexical methods, no matter how the hyperparameters get tuned.

SemDeDup takes a different approach entirely: represent each document as an embedding rather than a set of tokens. Specifically, it uses a pretrained 125-million-parameter OPT model and takes the last-layer embedding of each document's final token as its representation. Documents get clustered into groups of roughly similar content, and within each cluster, SemDeDup computes pairwise cosine similarity between all members. When two documents exceed a similarity threshold, the one embedded closer to the cluster centroid gets removed.

Applied to a subset of LAION, this approach removes half the dataset with minimal loss in downstream performance, which amounts to roughly halving training time for the same model quality. That is a substantial result, but it comes with a real caveat: SemDeDup, along with the related D4 method, has only been demonstrated on data derived from a large web scrape. Whether the same 50% redundancy rate and the same minimal-loss outcome holds on cleaner, domain-specific corpora, medical literature, legal filings, curated scientific text, has not been shown. Anyone applying this method outside of noisy web-scale data should validate on their own corpus before trusting the headline number to transfer.

Neural and fingerprint-chain approaches for noise-robust and containment-specific detection

Some corpora resist all of the methods above for a reason that has nothing to do with paraphrasing or semantics: noise. OCR errors, inconsistent formatting, encoding artifacts, and editorial variation between reprintings mean two documents can be functionally the same text while sharing very few exact n-grams. Lexical methods generally struggle under these conditions, because they depend on shingle-level agreement, and noise corrupts shingles unpredictably.

Silcock et al.'s NEWS-COPY benchmark was built specifically around this problem, using historical news wire reprints, where the same wire story would run in dozens of papers with different OCR quality and different manual re-typesetting. The dataset spans 27,210 documents with 122,876 labeled positive duplicate pairs, and it's a rare case where duplicate ground truth can be established from the historical record itself rather than requiring hand labeling across an entire massive corpus. The benchmark compares hashing and n-gram overlap methods against a contrastively trained bi-encoder and a combined bi-encoder-plus-cross-encoder re-ranking approach. The neural methods win clearly on this noisy data, outperforming hashing and n-gram overlap by a wide margin. The bi-encoder alone processes a 10-million-article corpus in a matter of hours on a single GPU, which suggests the usual worry about neural methods being too slow for corpus-scale work doesn't hold universally, at least at this scale.

FindMyText addresses a related but genuinely different question: not "how similar are these two documents" but "is this fragment contained inside that document." It builds on winnowing-based document fingerprinting, the same family of technique behind plagiarism detectors, but adds explicit detection of chains of matching fingerprints rather than just tallying the overall ratio of shared fingerprints. That chain detection lets it localize exactly where the shared fragment sits, rather than producing a single similarity score for the whole document pair, which is what makes it useful for copyright and licensing verification rather than bulk corpus cleanup. It scales to large web-crawled corpora through distributed, disk-based indexing, and on a new containment benchmark spanning ArXiv papers, Wikipedia, and general web content, it outperforms the alternative approaches tested.

Choosing between these tools is mostly a question of what's actually wrong with the corpus. Noisy historical or OCR'd text calls for a neural bi-encoder. Copyright or membership-inference questions against a fixed reference corpus call for something built around containment, like FindMyText. Bulk deduplication of a relatively clean web crawl is still, for most teams, better served by MinHash, simply because it's cheaper and the corpus doesn't need the extra noise tolerance.

Tooling and infrastructure for running deduplication at realistic scales

None of this matters if it can't run at the scale a real training corpus demands, and the tooling landscape here has moved fast.

DataTrove, Hugging Face's open-source data processing library, ships with native MinHash deduplication support and is probably the single most useful entry point for a team getting into this in 2025. On a corpus of 1 to 10 billion tokens (roughly 100 GB of text), a full MinHash pass on a 32-core CPU machine with 256 GB of RAM, signature generation, bucketing, clustering, and filtering combined, takes something like 4 to 8 hours, costing a modest sum on the order of the low tens of dollars on a typical cloud instance. That's a workable cost for most research teams, not just large labs.

text-dedup is built for a different point on the scale curve: billions of documents, with Jaccard thresholding, parallel execution, and low memory overhead built in. And for teams where even the LSH index itself becomes the bottleneck, LSHBloom replaces the standard MinHashLSH index with an array of independent Bloom filters, which gets 12 times the throughput of standard MinHash LSH while cutting disk footprint by a factor of 18. For a corpus large enough that index size and lookup speed start to dominate the cost of the whole pipeline, that's not a marginal improvement, it changes what's operationally feasible.