Duplicate Detection Fixtures

This sub-section exercises the near-duplicate page detector in the content_strategist scanner profile. Each fixture is labelled with its expected cluster behaviour so you can compare the actual summary.duplicates.clusters[] output against the spec.

Cluster A — Exact duplicates

Three pages with byte-identical body content. Chrome (nav/footer) and page titles differ. Expected: one cluster of size 3.

Cluster B — Near-duplicates, one-sentence delta

Three pages sharing the same body with one sentence paraphrased per page. Expected: one cluster of size 3 at the default threshold of 3.

Cluster C — Paragraph reordering

Two pages with the same three paragraphs in different order. SimHash is order-sensitive at shingle boundaries; clustering at threshold 3 is not guaranteed. Verify empirically.

Anti-cluster — Translated content

Same source text in English and Swedish. Different language tokens produce entirely different fingerprints. Expected: no cluster.

Anti-cluster — Short pages (under 30-word floor)

Pages with fewer than 30 words of body content. Excluded from fingerprinting entirely. Expected: cluster_id: null.

Anti-cluster — Same topic, different copy

Two pages about Clarity Analytics with substantially different body text (pricing vs. security). Expected: no cluster — proves the detector groups by copy similarity, not by topic.

Transitive cluster — Chain of 4

Pages T-1↔T-2, T-2↔T-3, and T-3↔T-4 are each within distance 3. T-1↔T-4 distance is likely above 3. Union-find should still merge all four into one cluster of size 4.

Chrome-only difference

Two pages with identical body content but very different nav/footer/aside blocks. Chrome is stripped before hashing. Expected: cluster of size 2.

Singleton control

A unique page about audit logging with no near-duplicates in this fixture set. Expected: cluster_id: null.