Monitoring YouTube without the Data API (and without the quota headache)

If you have ever tried to watch YouTube for new videos on a topic, you have probably met the YouTube Data API's quota system. A search.list call costs 100 units. The default daily allowance is 10,000 units. That is 100 searches a day — before you have read a single video's details.

For a one-off script that is fine. For "tell me when someone uploads a video mentioning my product", it is not. Polling five keywords every fifteen minutes would need 480 searches a day. You would be rate-limited before lunch.

You can apply for a quota increase. In my experience that is a slow process with an uncertain outcome, and it is a strange thing to need for reading public pages that any logged-out visitor can see.

So I built the boring alternative: read the public pages.

What YouTube actually gives you when logged out

Load youtube.com/results?search_query=... without a session and YouTube embeds the whole result set in the HTML as a JSON blob:

var ytInitialData = { /* ...several hundred KB... */ };

Everything the page renders is in there — video IDs, titles, channels, view counts, publish times, thumbnails. No API key involved.

The same is true for a channel's uploads tab.

The part that cost me an afternoon

My first parser walked the JSON looking for videoRenderer objects. It worked. It also silently returned 4 videos out of 14.

YouTube does not use one component. It uses several, and which one you get depends on the page and on whatever A/B test you have been bucketed into:

lockupViewModel is the newer one and it is shaped completely differently. Instead of videoId and title.runs[0].text, you get:

contentId                                                    -> video id
metadata.lockupMetadataViewModel.title.content                -> title
...metadata.contentMetadataViewModel.metadataRows[]
     .metadataParts[].text.content                            -> ["486K views", "12 days ago"]
contentImage.thumbnailViewModel.image.sources[-1].url         -> thumbnail

Once I handled both shapes, the same search page went from 4 videos to 17. Nothing had changed on YouTube's side — I had just been reading one of several boxes.

If you build this yourself, treat the component list as configuration, not as code:

CLASSIC_RENDERERS = ("videoRenderer", "gridVideoRenderer", "compactVideoRenderer")

When YouTube adds another one, you add a string.

The component I missed for a second time

Once I had lockupViewModel working I thought I was done. I was not.

A search for skincare returned 7 videos. The page actually held 39 more in a
fifth component, shortsLockupViewModel — YouTube Shorts. They are shaped
differently again:

entityId                                       -> "shorts-shelf-item-<videoId>"
overlayMetadata.primaryText.content            -> title
overlayMetadata.secondaryText.content          -> "2.7K views"
thumbnailViewModel.thumbnailViewModel.image    -> thumbnail (yes, nested twice)

Reading them took that search from 9 results to 46. For a lot of keywords Shorts
are the majority of what YouTube returns, so dropping them silently is not a
small omission.

They carry no publish time and no duration — the component simply does not
include either — so be honest about that in your output rather than inventing a
timestamp.

Two more things worth knowing

There is no exact publish time. Logged-out pages give you "12 days ago", never a timestamp. You can convert it, but be honest about what you have — I named the field published_ts_approx so nobody builds a sorting feature on top of a value that cannot support one.

Being blocked does not look like an error. When YouTube does not like your IP it returns HTTP 200 with a much smaller page, not a 4xx. Checking the status code tells you nothing. Checking the response size tells you everything:

if len(response.text) < 50_000:
    raise WallError("got a stub page, rotate the exit IP")

I lost real time to this before I noticed the pattern. Residential IPs and a retry that actually rotates the exit fixed it.

Why "only new" matters more than it sounds

The obvious design is: run it, get the current results. The problem is that monitoring means running it every fifteen minutes, and the same videos come back every time. You pay for them again, you filter them again, and your alerting channel repeats itself.

So the tool keeps a 7-day memory of video IDs keyed by the input set, and a scheduled run emits only what it has not seen. A run that returns nothing is the normal case, not a failure — and that is the behaviour you want from an alerting feed.

If you would rather not maintain this

I packaged it as an Apify Actor, because the scheduling, proxy rotation, dataset storage and webhook plumbing are not the interesting part:

YouTube Scraper | Monitor New Videos | No API Key

Keywords or channel handles in, new videos out, with the dedupe built in. There is a companion one for Meta Threads that works the same way:

Threads Scraper | No Login

Either way, I hope the component-shape table above saves you the afternoon it cost me.