Most write-ups on this subject are about technique. This one is about the three decisions you should make before you write any code, because in my experience every project that went badly went badly for a reason that was decided on day one and not noticed until much later.
I have built this kind of collection twice, for competitive analysis and for a partner-vetting workflow. Neither of them needed to touch anything behind a login, and I want to explain why that turned out to be the useful constraint rather than the limiting one.
Line one: the login wall is a boundary
A login wall is a statement about who the content is for. Treating it as an engineering obstacle to be routed around is the decision that puts a project on the wrong side of everything: terms of service, the platform's own detection, and in several jurisdictions the law.
So the first line is simply: if it requires an account to see, it is out of scope. Not "hard," not "for later." Out of scope.
I am not going to discuss techniques for getting past one, and I would be sceptical of any article that does. The interesting engineering question here is not how to see more. It is how much you can actually do with what is openly published, and the honest answer is: considerably more than people assume before they check.
This constraint also has a practical benefit that is easy to miss. A pipeline built only on openly available data does not break when authentication changes, does not require credential management, and does not put an account at risk. Mine has survived two platform changes that took down colleagues' authenticated collectors.
Line two: public does not mean unrestricted
The second line is the one developers get wrong most often, and it has nothing to do with access.
Data being publicly visible says nothing about whether you may store it, for how long, or what you may do with it. In the EU and UK, information about an identifiable person is personal data whether or not they published it themselves, and a public profile is very obviously about an identifiable person.
That does not make collection impossible. It makes it something you have to have a reason for, and the reason has to be written down before rather than reconstructed afterwards.
Three practical consequences I build in from the start.
Have a stated purpose, in a document, before collecting. "Competitive benchmarking of brand accounts in our category" is a purpose. "We might want it later" is not, and it is the one that will not survive any subsequent conversation with a lawyer.
Set a retention period and enforce it in code. Mine is 90 days for post-level records and indefinite for aggregate statistics, because aggregates about a category stop being about a person once they are aggregated. Enforce it with a scheduled deletion job, not a calendar reminder.
Distinguish business accounts from personal ones. An account operating openly as a business, with a category and a contact method it has chosen to publish, is a different proposition from a private individual's account that happens not to be locked. I collect the first and exclude the second, and that single filter removes most of the genuinely uncomfortable cases.
Line three: rate limits are a contract
The third line is about behaviour rather than scope.
A rate limit is the platform telling you what load it is prepared to serve. Backing off when you hit one is the correct response. Engineering around one is a decision to take more than you were offered, and it is also, practically, how a collection project ends.
My rules are unglamorous. Sequential rather than parallel. Delays that are generous rather than minimal. A hard daily cap on total requests set well below anything I have ever seen enforced. Respect for anything the site's own robots directives express.
Slow collection is fine. Almost nobody actually needs this data in real time. The competitive analysis I run is a weekly job, and the difference between finishing in ten minutes and finishing in four hours is invisible to everyone including me.
The one thing I would push back on is the idea that this is only about avoiding a block. Load you generate is load someone else pays for. That framing has been more useful to me for making good decisions than any risk calculation.
What is actually available inside those lines
Data Openly available How I treat it
Handle, display name, bio Yes, on public accounts Collect; the bio is free-text and may contain personal detail, so it is not exempt from line two
Follower, following, post counts Yes Collect; pure numbers, lowest sensitivity
Category and published contact on a business profile Yes, when the account chooses to show it Collect for business accounts only; this is the clearest legitimate-interest case
Post caption, timestamp, media type Yes, on public accounts Collect; captions are free text and get the same treatment as bios
Public like and comment counts Yes Collect; this is what engagement rate is computed from
Hashtags, mentions, tagged accounts in a post Yes Collect the tags; mentions identify third parties who did not post, so I aggregate rather than store
Hashtag feeds and post counts Yes Collect; the least personal data on this list and often the most useful
Individual commenter identities Sometimes visible Do not store. High sensitivity, low analytical value, easy to skip
Stories, DMs, follower lists, private accounts No Out of scope, permanently
Impressions, reach, saves, audience demographics No Not observable from outside. Anyone offering these for accounts they do not own is describing data they cannot have
The bottom row deserves emphasis because it comes up in every requirements conversation I have had. Someone will ask for a competitor's reach or their audience breakdown. Those numbers exist only in that account's own analytics, and no external collection produces them. The correct answer is that the question cannot be answered rather than that it is difficult.
What the open data is actually good for
The obvious use is engagement rate, computed from public likes and comments against follower count on recent posts. It is a crude measure and it is still the single most useful thing available from outside, because a follower count on its own tells you nothing about whether anyone is there.
Anyone can compute it by hand for one account. The reason to build or buy anything is the set: a category is a benchmark and one account is a data point. Tools built for this operate the same way, and the ones worth using are explicit that they scrape Instagram posts without login, covering public profiles, posts and hashtag feeds and nothing beyond them. AllyHub computes the engagement figure from the public likes and comments rather than implying access to anything private, which is the disclosure I would look for in any option here.
The less obvious use, and the one I have got more from, is hashtag ecosystem mapping. Which tags co-occur, which accounts sit at the intersections, how a topic's vocabulary shifts across quarters. It involves almost no personal data, it is genuinely hard to get any other way, and it answers strategy questions rather than vanity ones.
The test I apply to every field
Before a field goes into the schema, three questions.
Can I see it without an account? Would I be comfortable if the account owner read my purpose statement? Is there a use for it already written in that statement?
Two yeses and a no is a no. The third question kills the most fields and kills them for the right reason. Anything collected because it happened to be there is precisely the data that makes a dataset harder to defend and that nothing downstream ever reads.
I dropped four fields from my first schema on this test and have not once wanted them back. The instinct to collect broadly and decide later is the same instinct that produces the retention problem in line two, arriving a step earlier.
Things I have declined to build
A request to identify individual commenters and enrich them against an email list. Technically feasible from visible data, and I turned it down. Building an identity graph of individuals from social activity is a different category of thing from benchmarking brand accounts, regardless of every item in it being visible.
A request to monitor a specific person's account activity over time. Same data types as a competitive brand monitor, entirely different in what it is. If replacing the account name with a person's name in the project description makes it sound like surveillance, it is surveillance.
A request to reconstruct historical data from before collection started by inferring it. The honest answer is that you cannot, and an inferred number in a dataset of measured ones is worse than a gap because nothing downstream will remember it was invented.
Practical setup
Write the purpose statement first, in the repository, next to the code. Two paragraphs. It will feel like ceremony and it is the artifact that makes every later decision easy.
Log what you collected and when, at the field level. When someone asks whether you hold data on X, you want that to be a query and not an investigation.
Separate collection from analysis with a storage boundary. Collection writes raw records with a timestamp and a retention flag. Analysis reads aggregates. That separation is what makes a 90-day deletion policy implementable rather than aspirational.
And build the exclusion list before the include list. Mine names account types I do not collect, and it has been more useful for keeping the project on the right side of things than any positive specification.
Put a calendar entry on re-reading the platform's terms twice a year. They change, the changes are not announced to you, and "it was allowed when we built it" is not a position anyone wants to be arguing from. Twice in three years a re-read has caused me to narrow something, both times cheaply because the collection was narrow to begin with.
Where this leaves you
The three lines rule out a lot: anything authenticated, anything about private individuals, and anything that treats a rate limit as a challenge.
What is left is still a substantial and genuinely useful dataset, and it has the property of being defensible, which the alternative does not. I have never wanted the excluded data badly enough to reconsider, and the two projects that survived contact with a compliance review survived because these decisions were made on day one rather than argued about in month four.
