Can I scrape this site? Start with access, not code.
A practical access checklist for public web data: logins, robots, official APIs, rate limits, personal data and the difference between reachable and permitted.
The short answer
A page being reachable does not mean bulk collection is permitted. Check four things before choosing a tool: whether the data needs a login, what the site terms say about automated access, whether an official API exists, and whether the output contains personal data. The answer can differ by page, field and jurisdiction.
Catalogue and method checked 2026-11-12. See how tools are scored.Most scraping failures are access failures, not parser failures. The site redirects to a login, the API requires a partner agreement, the list stops at a result ceiling, or the data includes personal information with its own rules. Check those constraints before comparing Actor prices.
Four access questions
Start with the route to the page. Is it public, login-gated, partner-only or retired? Then read the terms that apply to automated access and reuse. Next, check whether an official API covers the exact object you need. Finally, classify the output: business data, public content, personal data or something mixed.
- Public: visible without an account, but still subject to terms, robots and copyright.
- Login-gated: do not bypass the login or use another person's session.
- Partner-only: use the official programme or a licensed provider.
- Retired: check the deadline and the replacement route before building.
Reachable is not the same as permitted
Robots.txt is a crawl directive, not a complete legal analysis. Site terms can restrict automated access, bulk export or reuse even where a page is public. Copyright, database rights and privacy law sit on top of the contract.
The practical rule is to collect the minimum public data needed for the task, keep the source and check date, and do not bypass a login, CAPTCHA or access control. This is general information, not legal advice.
When an official API is the better route
An official API is usually the clearest route when it covers the object and the volume you need. It may cost more per call, but it comes with documented limits, authentication and a support path. A public-data Actor can fill gaps where the official API is absent, narrow or partner-only.
Use the target page to compare both routes side by side. The API status, limits and terms links are kept on the page, so the access decision is separate from the ranking.
Questions people ask
Does robots.txt decide whether scraping is legal?
No. Robots.txt is a crawl directive. It does not replace the site terms, copyright, database rights, privacy law or a contract. Treat it as one input, not a legal conclusion.
Can I scrape a site if I can see the page without logging in?
Not automatically. Public visibility still leaves the terms, copyright and privacy rules in place. Bulk collection and reuse can be restricted even when a single page is visible.
What should I do when a source closes public access?
Use the documented replacement route, export any data you are entitled to keep, and update the workflow rather than retrying a blocked endpoint. The error and API pages on this site track dated changes.
Can an Actor bypass a login for me?
No. The catalogue route is for public data. Do not use a tool to bypass a login, CAPTCHA or access control. Check the source terms and use an official or licensed route where one exists.
Sources
These sources support the general method on this page. A specific tool's price, schema or access route is on its target page with the check date.
- Google Maps Platform terms
- LinkedIn user agreement
- Reddit public access update
- FTC CAN-SPAM guide
- Data sources and check policy
- Ranking method
Pick the website you need data from
Each website has a route map: every catalogue Actor for it, the fields each one returns, the price per 1,000 results and the official API beside them. Or search all 394 Actors by the field you need.