I build devdigest, a daily tech digest. It pulls from RSS feeds, has a model read each excerpt to categorise and score it, writes an original summary, and emails subscribers a list of titles, summaries and links back to the original articles. Full article text is never reproduced or stored beyond a short-lived per-run cache. To read anything, you click through to the publisher.
Before charging money for that, I wanted to know whether any of my sources explicitly banned it. So I read the actual Terms of Service for all of them. Not the privacy policy, not a summary, the real current terms, quoting the clause or explicitly recording "no ToS found."
I expected the answer to track company size. Big corporate sites run their own RSS feeds, press APIs and developer-relations teams, so I assumed they would be relaxed. Small scrappy publications felt like the ones who would care.
I had it exactly backwards.
The split
The most restrictive terms came from VC-data companies and large corporate infrastructure blogs. Verbatim:
Crunchbase News bans using content "to train models (including generative artificial intelligence technologies)," and separately bans anything that "'Crawls,' 'scrapes,' or 'spiders' any page, data, or portion of" the content.
HPCwire/AIwire bans any "robot, spider, or other automatic device" without prior written permission, plus "any form of data extraction or data mining, or other commercial exploitation of any kind." The most specifically targeted restriction I found anywhere.
TechRadar (Future plc): "You shall not conduct, facilitate, authorise or permit any text or data mining or web scraping... for any purpose, including the development, training, fine-tuning or validation of AI systems or models."
Cloudflare's blog bars automated bots from "scrape, data mine, copy, or use the materials or content on this website for developing, training, fine-tuning, or otherwise contributing to or improving a machine learning model or artificial intelligence (AI) system."
Sifted (§7.6), the European startup publication: "You must not conduct, facilitate, authorise or permit any text or data mining or web scraping in relation to our Platform... for any purpose, including the development, training, fine-tuning or validation of AI systems or models."
Now the other direction, from the sites I assumed would be precious:
InfoQ: "We permit the posting of a summary and then a link back to the InfoQ landing page." That is my exact pipeline, described approvingly, in their terms.
MIT News: its Terms of Use state it "offers RSS feeds for syndication purposes."
GitHub: "these Terms do not restrict lawful access to or use of the contents of public repositories by third parties."
arXiv: the API Terms of Use explicitly names RSS-based discovery and notification tools as a permitted use case, in those words.
TechCrunch maintains dedicated RSS terms, separate from its general ToS, explicitly permitting display of feed content with attribution and a link to the full article.
Engineering blogs, university news offices and syndication-native sites were mostly fine with it. The well-lawyered commercial sites were not.
My best guess at why: the restrictive ones have their own data-licensing business to protect, or expect to. A clause banning AI training is an asset if you intend to sell that access later. The permissive ones live on distribution. Being read and linked to is the product, so the feed is a front door rather than a leak.
The part that actually matters: three ways this goes wrong
The size heuristic being backwards was interesting. What was useful was discovering that the obvious check is wrong in three different directions.
1. The ToS is clean and robots.txt bans you.
I originally cleared Towards Data Science. It is Medium-hosted, and Medium's Terms of Service contain no RSS, scraping, commercial-use or AI-training restriction. Clean read.
I had not checked robots.txt. Medium's explicitly disallows ClaudeBot, GPTBot and every other major AI crawler by name. My pipeline runs on OpenAI's model, so GPTBot's presence in that list was not an academic detail. I removed the source.
Checking the terms and not robots.txt gave me a clean answer to the wrong question.
2. The site publishes an RSS feed and its ToS bans scraping anyway.
Sifted surfaces an RSS feed link on its own homepage. Its Terms of Use, quoted above, ban text and data mining and separately define and prohibit any "'robot', 'bot', 'spider', 'scraper' or other automated device."
A published feed is not a licence. It is extremely easy to read the existence of /feed as permission, because for most of the web's history that is roughly what it meant. It does not mean that now, and the terms are the binding document.
3. The scraping clause exists but is scoped somewhere else entirely.
TechCrunch has a "robots/spiders/scrapers" clause. On a keyword search it looks like a red flag. Reading it in context, it is scoped to user-submitted content, comments and forums, not published articles, and the separate dedicated RSS terms permit exactly what I was doing.
If I had grepped for "scraper" and moved on, I would have dropped a source that explicitly allows this. False positives cost you real sources.
Boilerplate is not signal
The single most useful methodology note, and the one that took longest to arrive at.
Nearly every commercial media ToS carries some version of "all rights reserved, no reproduction without permission." It is close to universal. Treated as a prohibition, it bans what every RSS reader, every feed aggregator and every newsletter on earth does daily.
So it cannot be the test. If your rule flags everything, it has told you nothing.
What I ended up looking for is a clause that specifically names automated access, bots, scraping, RSS or feed reuse, or (increasingly common since 2024) AI training and summarisation. Those describe the actual pipeline. Generic anti-plagiarism language does not.
The rule I settled on: exclude only on an explicit stated prohibition. Never infer a ban from silence. Several sources had no discoverable terms at all - Meta's AI blog has no ToS page whatsoever, Simon Willison's blog has a bare copyright symbol and nothing else. Silence is not permission, but it is also not a prohibition, and inventing one costs a good source for nothing.
What I did about it
Eight sources came out over the audits: Crunchbase News, HPCwire/AIwire, TechRadar and Cloudflare's blog on explicit clauses; Ars Technica and Wired on Condé Nast's posture, including active litigation against an AI company over scraping, where I could never get a verbatim read; Towards Data Science on the robots.txt finding; Sifted once §7.6 turned up in a later pass.
The list has since grown past 120 sources and the audit runs on a schedule rather than once, because terms change. The 2024-onward AI-training clauses are mostly new language. A source cleared in 2023 is not cleared now, and several of the clauses above carry recent dates.
None of these sites did anything wrong. Writing a restrictive ToS is entirely a publisher's prerogative, and given how much unlicensed scraping has funded model training, the defensive drafting is easy to understand. I quote them here because the terms are public and because "check for yourself" is useless advice without showing what the check actually looks like.
Caveats
I am not a lawyer and this is not legal advice. This is one developer reading terms carefully before shipping, which is a lower bar than a real IP review and a considerably higher one than not looking.
Two things I would tell anyone building something that reads other people's content automatically:
Check the terms and robots.txt, separately, and treat them as independent. They disagree, in both directions.
And check the sites you assume are fine. My entire mental model of who would object was wrong, and the only reason I know that is that I read all of them instead of the ones I expected to be a problem.
