It all started when I kept asking this question every day:
Wouldn't it be easy if the crawler can just look at the URL and tell whether its a structure URL pattern or an Identifier(hash, slugs, IDs)
Painpoints
- Pattern matching failed badly for distorted Page URLs where the vocabulary was simply different and the slug patterns were inconsistent
- Without pattern matching, we couldn't really identify a large content hub that produces tons of articles/products/blogs…etc This makes crawling miserable of large fan-out pages! The only way was to crawl everything!
- Trie was built in to model a site so we can reasonably determine page structures. But it all went backfired when we entered large Ecom style fan-out sites who simply don't follow Trie URL structure 🙂
- Deterministic rules helped with Hash detection, numbers, but failed badly with "content slugs". They are always not too long, don't have atleast 2 hyphens, mix of alpha and numbers. Could not really distinguish them based one generalised rule. Kept going back and forth between heuristic rules!
Iteration 1
I was ready to train a text model that uses some kind of clustering like K-means or Naive Bayes. A small prototype was made based on Naive bayes , used dev URLs as a data test.
The Good: It actually helped identify positive paths better
The Bad: But prod's dataset was too wide. The generalisation needed more dataset and a better pipeline.
Realising I'm only a noob at building models, gave up and proceeded with next approach.
Iteration 2
The prototype worked well for already generalised well defined keywords. It didn't help with the site specific hubs. I won't blame the model, who the heck is going to know that application/marine could be a large content hub!
So I added a local vocabulary on top of the generalised model and let the logic use this as a fallback when the generalised model could not classify it.
No good, just bad: Although it looked like a good idea, unfortunately what could have been general on the dataset was not the general case in most sites. Similarly what was rare in the general set was again not rare at all for this set.
Tried adjust scores from global+local, but again its a goose hunt to find the right attribution!
Iteration 3
Go all in on local vocab. Every site is a unique beast, all I care about is identifying large fan-out with sparse URL patterns to ban them.
The Good: This performed better than both the previous iterations and was able to identifying structural vs Identifier reliably purely on the sites on vocab.
The Bad: We lost the predefined judgement which generalised model gave us, it took the crawler time to understand site structure. So it was bucketing similar URLs into separate groups rather than one.
Iteration 4
Multiple ideas to solve the problem:
I actually thought of a fake crawl to learn the site and then do the actual crawl. But bad too costly and time consuming
How about we limit to 100 page fetches to build vocab then use the judgement. Again a heuristic ticking bomb to blow out.
Instead the better approach was to build resilience. Like how we humans think we start with a guess, consume enough volume so adds more signals to the clusters and makes them distinct right!
So we already had a checkpoint program that runs every 15 fetches. First we adjust that to 50 and re-pattern all the URLs using the updated vocab(a bit costly on the DB! but worth it) . This helped with the buckets convergence. And to top it off we have guards in-place so we quantify "Rarity" vs "Structural".
The rare keywords in a site might not necessarily be 1,2,3..etc. It depends on the site itself, similarly the structural keywords cannot be 40,50...etc. In the beginning, we have no way to guess. But as the crawl progresses we recalibrate these, given the buckets get distinct as we collect more vocabulary. The more we crawl, the more distinct the structural and rare keywords become, and we re-pattern them based on our new finding.
TLDR;
Heuristic always fails, there is always a black swan. Building resilience as we gather more evidence and adapting on the fly helps. This exercise made me realise that. It also helped me realise its possible to do without complicated models…
