Duplicate Content Checker: Compare Page Similarity

Two URLs can rank for nothing useful when they say the same thing with different slugs. Marketing calls it "syndication." Search engines call it a pick-one problem. You do not need a lecture on duplicate content theory to fix it on your own site. You need to know which pairs are too similar and what text they share.

This guide walks through a practical order: pick candidates, compare body copy, decide consolidate vs differentiate, then wire canonicals or redirects. The matching tool is the Duplicate Content Checker on Scrawl: 2–5 URLs or pasted text blocks, main-body extraction, pairwise similarity, shared passages, and an 80%+ near-duplicate flag. It compares only what you enter, not the web.

Start with a short list, not a full crawl

Duplicate panic often starts with someone exporting every URL from analytics. That is the wrong first move. Start from:

• Pairs you already suspect (two service area pages, two SKU variants with the same spec sheet)
• URLs that swapped positions in Search Console after a template change
• Drafts from writers that "sound familiar"

Put those URLs in the checker. If the list grows past five, run multiple batches grouped by template. Screaming Frog includes near-duplicate detection in its paid licence when you need sitewide coverage. This browser tool is a quick first pass on a handful of pages.

What the checker actually compares

For URL mode the tool:

1. Fetches each page (initial HTML, no JavaScript rendering)
2. Removes nav, header, footer, scripts, and similar chrome
3. Extracts visible body text from article, main, or body
4. Tokenizes words and builds shingles for Jaccard similarity
5. Surfaces the longest shared word sequences between each pair

Paste mode skips the fetch and compares text you supply. Use it when staging is blocked, when you only care about the article field from a CMS export, or when global header HTML would swamp the score.

Important: similarity here is lexical overlap on body text. It does not judge search intent, hreflang, or whether Google chose a different canonical. It answers: "How much of the visible copy is the same?"

How to read the similarity percentage

Jaccard similarity on word shingles rewards repeated phrases. Expect higher scores when:

• Product descriptions reuse the same bullet list with one word changed
• Location pages share a boilerplate paragraph
• Blog posts start with the same disclaimer block

Treat 80% or higher as a near-duplicate flag in this tool. That is a triage threshold, not a penalty score from Google.

Lower scores can still matter. Two pages at 55% might share one long spec table while differing elsewhere. Read the shared passages block: that is where consolidation work starts.

Decide: merge, differentiate, or canonicalize

Once you have a hot pair, pick a strategy:

Merge when both URLs serve the same intent and neither has links or history worth keeping separate. Pick the keeper URL, 301 the other, update internal links.

Differentiate when both URLs should exist (two products, two cities) but copy was lazy. Rewrite the overlapping sections so each page has unique facts: pricing, availability, local proof, specs.

Canonicalize when both must stay live for users but only one should collect signals. Set a self-referencing canonical on the primary URL and a canonical pointing to it on the variant. Confirm with the Canonical Checker.

Title and meta overlap is a related but separate issue. Run the Duplicate Title & Meta Finder when SERP snippets look identical even after body copy is fixed.

Pair with other Scrawl checks

| Situation | Tool |

|-----------|------|

| Staging vs live body after deploy | Diff Checker |

| Expected title/H1/description vs live | SEO Content QA Checker |

| Cannibalizing titles across many URLs | Duplicate Title & Meta Finder |

| Wrong canonical on a near-duplicate pair | Canonical Checker |

Worked example: twin location pages

You have /plumbers-denver and /plumbers-boulder with the same three paragraphs and only the city name swapped.

1. Paste both URLs into the Duplicate Content Checker.
2. If similarity is above 80%, open shared passages and note the boilerplate chunk.
3. Rewrite each page with city-specific proof: neighborhoods served, licenses, photos, reviews.
4. Re-run the checker on the updated URLs. Aim for clearly lower similarity and shorter shared passages.
5. If one page is obsolete, 301 it instead of keeping thin twins.

Mistakes that waste an afternoon

• Comparing pages with different templates (blog vs product) and expecting a low score when both share site-wide chrome.
• Ignoring JavaScript-only body text. If the article loads after hydration, fetch mode will under-read. Paste the rendered HTML from a one-off save or compare in a rendering crawl.
• Canonicalizing without changing copy when both pages should rank for different queries. Google may ignore the canonical if content is too similar and user signals disagree.
• Assuming this tool searched competitors. It did not. Competitor copies are a legal and manual research problem.

FAQ

How do I check two pages for duplicate content?

Use the Duplicate Content Checker with two URLs or pasted bodies.

What does near-duplicate mean here?

80%+ Jaccard similarity on word shingles after body extraction.

Does it crawl my site?

No. Only the list you provide.

Will footers inflate scores?

Often. Paste mode or same-template comparisons reduce noise.

When do I need Screaming Frog?

When you need near-duplicate reports across a full crawl under its paid licence.

Compare your pages now

Add your candidate URLs or pasted text to the Duplicate Content Checker, read pairwise scores and shared passages, then fix with merges, rewrites, or canonicals as the business case requires.