I Built an Agent That Found Real Duplicate Content in My Production SEO Database

I run RankinSEO, an AI-powered SEO article platform for trades businesses (roofing, HVAC, plumbing, solar, electrical). Over the past few months I've manually hunted down duplicate content in my Supabase database by eyeballing a sorted table — tedious, error-prone, and something I kept meaning to automate.

For the Sanity Challenge, I finally built that automation as a real agent.

What I built

Migrated a curated sample of my live content (5 articles per trade) from Supabase into Sanity's Content Lake as structured article documents
Wrote a GROQ-powered agent that queries the content and analyzes title similarity to flag potential duplicates
Ran it against my full 130-article production table to see if it would re-discover issues I'd already found manually

The GROQ query at the core of it

*[_type == "article"] {
  title,
  slug,
  trade
} | order(trade asc)

The agent then normalizes each title (strips stopwords and punctuation) and computes word-overlap similarity between every pair, flagging anything above a 40% threshold.

What it found

Running against my full dataset, the agent correctly re-discovered several duplicate clusters I'd already manually identified and fixed back in September — like "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026" (75% overlap) and "Commercial Roofing" vs "Commercial Roofing Company" (67% overlap).

It also honestly surfaced a limitation: a Calgary-localized article with a typo in its slug ("roofing-contrators-near-me") got flagged as similar to several generic "near me" articles — purely because of shared words like "roofing" and "near me" — even though the actual content and intent were genuinely different. Pure word-overlap similarity can't distinguish "same topic, different angle" from "actual duplicate." That's a real constraint worth knowing before trusting this kind of tool blindly.

Why this matters

This isn't a toy demo — it's solving a problem I actually have, using my actual production content. The agent won't replace human judgment on which "duplicates" to merge (context still matters), but it turns a 20-minute manual scan into a 3-second automated first pass.