Study · updated
What it takes to get data from 172 major websites
39 of the 172 websites offer a public official API (23%). The rest limit theirs to approved uses (47), keep it for partners (36), have none (45) or retired it (5).
Asked once with a plain HTTPS request, 59 of them (34%) refused to hand over their robots.txt or their terms page. Of the 130 robots.txt files that could be read, 14 disallow every path for crawlers without their own rules, and 63 single out AI crawlers by name.
39public official API 59refused a plain request 14robots.txt disallows every path 63robots.txt names AI crawlers
Is there an official API?
One square per website, from each website's own developer documentation, dated on its page.
- Public API 39
- Limited access 47
- Partner only 36
- No open API 45
- Retired 5
Does a plain request get an answer?
Once a week the site asks each website for two public files, its robots.txt and its terms page, the way a simple script would: one HTTPS request each, no browser, no cookies. A refusal is an error status such as 403 Forbidden or 429 Too Many Requests.
- Both files answered 113
- One file refused 44
- Both files refused 15
robots.txt: 137 answered, 22 refused, 4 not found, 9 no answer. Terms page: 111 answered, 52 refused, 4 no answer, 5 not read.
What robots.txt says to crawlers without their own rules
The rules under User-agent: *, the group every crawler falls back to. 5 files also ask for a crawl delay.
- No disallow rules 5
- Some paths disallowed 111
- Every path disallowed 14
- Could not be read 42
AI crawlers named in robots.txt
Of the 130 robots.txt files read, how many give each AI crawler or AI-training token its own rules (usually to keep it out).
No public API, and every path disallowed
10 websites have no public official API and a robots.txt that disallows every path for crawlers without their own rules: Facebook, Google News, IMDb, LinkedIn, Naver, Quora, Reddit, TikTok Ads Library, Trustpilot and ZipRecruiter. Each of their pages lists every route that does exist, with a link to the website's own terms.
By kind of website
| Kind of website | Websites | Public API | No open API | Disallow every path | Refused a plain request | Name AI crawlers | Open-source scrapers |
|---|---|---|---|---|---|---|---|
| Ecommerce | 36 | 4 | 9 | 0 | 13 | 8 | 22 |
| Jobs | 20 | 6 | 8 | 1 | 8 | 10 | 4 |
| Real estate | 13 | 0 | 9 | 0 | 7 | 3 | 7 |
| Social network | 10 | 3 | 1 | 4 | 3 | 5 | 7 |
| Travel | 10 | 0 | 1 | 0 | 5 | 5 | 4 |
| Sports and gaming | 6 | 1 | 3 | 0 | 2 | 1 | 3 |
| Local and place data | 5 | 3 | 0 | 0 | 0 | 1 | 1 |
| News and publishing | 5 | 1 | 2 | 1 | 1 | 3 | 3 |
| Video | 5 | 3 | 0 | 0 | 0 | 2 | 5 |
| Ads and transparency | 4 | 0 | 2 | 1 | 2 | 0 | 1 |
| Finance and markets | 4 | 2 | 1 | 0 | 2 | 1 | 2 |
| Media and entertainment | 4 | 0 | 0 | 1 | 0 | 4 | 4 |
| Research and reference | 4 | 3 | 1 | 0 | 0 | 0 | 4 |
| Search results | 4 | 1 | 0 | 1 | 0 | 0 | 3 |
| SEO and web analytics | 4 | 1 | 0 | 0 | 1 | 1 | 2 |
| Other | 38 | 11 | 8 | 5 | 15 | 19 | 16 |
"No open API" counts websites with no official API at all; limited, partner-only and retired APIs are in the CSV. "Other" groups kinds of website with fewer than four websites.
Every website
Sort by any column. Each name opens the website's page with its robots.txt rules, terms clause, court cases and every route to its data.
| 2GIS | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
|---|---|---|---|---|---|---|
| 99acres | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 1 | ||
| Ahrefs | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Airbnb | No open API | Some paths disallowed | 10 | robots.txt answered, terms answered | 0 | |
| Alibaba.com | Partner only | Some paths disallowed | 2 | robots.txt answered, terms answered | 0 | |
| AliExpress | Partner only | Some paths disallowed | 6 | robots.txt answered, terms answered | 5 | |
| Allegro | Limited access | Some paths disallowed | 1 | robots.txt answered, terms refused 403 | 0 | |
| Amazon | Limited access | Some paths disallowed | 21 | robots.txt answered, terms refused 403 | 1 | 4 |
| Apartments.com | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 0 | ||
| Apple App Store | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Ashby job boards | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| AutoScout24 | Partner only | Some paths disallowed | 5 | robots.txt answered, terms answered | 1 | |
| Baidu | Public API | Every path disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Bayt | No open API | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 0 | |
| Bayut | No open API | Some paths disallowed | 0 | robots.txt answered, terms refused 503 | 0 | |
| Best Buy | Public API | Could not be read | robots.txt no answer, terms answered | 0 | ||
| Bilibili | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 3 | |
| Bing | Retired | Some paths disallowed | 0 | robots.txt answered, terms not read | 1 | |
| Bing Maps | Retired | Could not be read | robots.txt not found, terms not read | 0 | ||
| BizBuySell | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 1 | ||
| Bluesky | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Bol.com | Partner only | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 0 | |
| Booking.com | Partner only | Some paths disallowed | 0 | robots.txt answered, terms answered | 1 | |
| Brazil CNPJ registry | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Built In | No open API | Some paths disallowed | 4 | robots.txt answered, terms answered | ||
| Capterra | No open API | Some paths disallowed | 1 | robots.txt answered, terms refused 403 | 0 | |
| Coupang | Partner only | Could not be read | robots.txt answered, terms answered | 3 | ||
| Coursera | Limited access | Some paths disallowed | 9 | robots.txt answered, terms answered | 1 | |
| Craigslist | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 1 | 7 |
| Crexi | No open API | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 0 | |
| Crunchbase | Limited access | Some paths disallowed | 21 | robots.txt answered, terms answered | 2 | |
| Dailymotion | Limited access | Some paths disallowed | 0 | robots.txt answered, terms not read | 1 | |
| DeviantArt | Public API | Some paths disallowed · delay 1 s | 0 | robots.txt answered, terms no answer | 4 | |
| DEX Screener | Public API | Could not be read | robots.txt refused 403, terms answered | 3 | ||
| DoorDash | Partner only | Could not be read | robots.txt refused 403, terms answered | 0 | ||
| eBay | Public API | Some paths disallowed | 13 | robots.txt answered, terms answered | 2 | |
| ESPN | No open API | Some paths disallowed | 8 | robots.txt answered, terms answered | 0 | |
| Etsy | Public API | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 1 | |
| Eventbrite | Limited access | Some paths disallowed | 1 | robots.txt answered, terms refused 403 | 0 | |
| Expedia | Partner only | Some paths disallowed | 7 | robots.txt answered, terms refused 429 | 1 | |
| Limited access | Every path disallowed | 6 | robots.txt answered, terms refused 400 | 4 | 1 | |
| FINN.no | Partner only | Some paths disallowed | 11 | robots.txt answered, terms answered | 0 | |
| Fiverr | Limited access | Some paths disallowed | 11 | robots.txt answered, terms answered | 6 | |
| Flipkart | Partner only | Some paths disallowed | 1 | robots.txt answered, terms answered | 1 | |
| Florida Sunbiz | Public API | Could not be read | robots.txt refused 403, terms refused 403 | 0 | ||
| Fotocasa | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Freelancer.com | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| G2 | No open API | Some paths disallowed | 8 | robots.txt answered, terms answered | 0 | |
| GetYourGuide | Partner only | Could not be read | robots.txt refused 403, terms answered | 0 | ||
| GitHub | Public API | Some paths disallowed | 9 | robots.txt answered, terms not read | ||
| Glassdoor | No open API | Some paths disallowed | 15 | robots.txt answered, terms refused 403 | 5 | |
| Goodreads | Retired | Some paths disallowed | 2 | robots.txt answered, terms answered | 2 | |
| Google Ads Transparency Center | No open API | Could not be read | robots.txt no answer, terms answered | 0 | ||
| Google Flights | Partner only | Could not be read | robots.txt answered, terms answered | 4 | ||
| Google Maps | Public API | Could not be read | robots.txt not found, terms answered | 3 | ||
| Google News | No open API | Every path disallowed | 7 | robots.txt answered, terms answered | 7 | |
| Google Patents | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | |
| Google Play | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 5 | |
| Google Scholar | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 7 | |
| Google Search | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | 4 |
| Google Shopping | Limited access | Could not be read | robots.txt no answer, terms answered | 2 | ||
| Google Trends | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 4 | |
| Google Workspace Marketplace | Limited access | Could not be read | robots.txt no answer, terms answered | 0 | ||
| Greenhouse job boards | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| GulfTalent | No open API | Some paths disallowed | 16 | robots.txt answered, terms refused 403 | 0 | |
| Hacker News | Public API | Some paths disallowed · delay 30 s | 0 | robots.txt answered, terms answered | 1 | |
| Handshake | Limited access | No disallow rules | 6 | robots.txt answered, terms answered | 0 | |
| Hepsiburada | Partner only | Could not be read | robots.txt refused 403, terms refused 403 | 0 | ||
| Himalayas | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Hostelworld | Partner only | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Houzz | Partner only | Some paths disallowed | 7 | robots.txt answered, terms answered | 0 | |
| Idealista | Partner only | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 4 | |
| IKEA | No open API | Some paths disallowed | 1 | robots.txt answered, terms answered | 0 | |
| IMDb | Limited access | Every path disallowed | 2 | robots.txt answered, terms answered | 3 | |
| Immowelt | No open API | Some paths disallowed | 7 | robots.txt answered, terms refused 403 | 0 | |
| ImportYeti | Public API | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 0 | |
| Indeed | Partner only | Some paths disallowed | 23 | robots.txt answered, terms answered | 7 | |
| IndiaMART | Limited access | Some paths disallowed | 8 | robots.txt answered, terms refused 403 | 0 | |
| Limited access | Could not be read | robots.txt answered, terms refused 400 | 3 | 2 | ||
| JobStreet | Limited access | Could not be read | robots.txt not found, terms answered | 1 | ||
| Kayak | Partner only | Some paths disallowed | 1 | robots.txt answered, terms answered | 0 | |
| Lever job boards | Public API | No disallow rules · delay 1 s | 12 | robots.txt answered, terms answered | 0 | |
| Partner only | Every path disallowed | 19 | robots.txt answered, terms answered | 3 | 1 | |
| LinkedIn Ad Library | No open API | Could not be read | robots.txt refused 403, terms answered | 0 | ||
| LinkedIn Jobs | Partner only | Could not be read | robots.txt not found, terms answered | 5 | ||
| Lowe's | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 0 | ||
| Medium | Limited access | Some paths disallowed | 7 | robots.txt answered, terms refused 403 | ||
| Meetup | Limited access | Some paths disallowed | 1 | robots.txt answered, terms refused 403 | 0 | |
| MercadoLibre | Public API | Some paths disallowed | 7 | robots.txt answered, terms answered | 2 | |
| Mercari Japan | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Meta Ad Library | Limited access | Could not be read | robots.txt refused 403, terms refused 400 | 1 | ||
| MLB.com | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Myntra | Partner only | Could not be read | robots.txt answered, terms no answer | 0 | ||
| Naukri | No open API | Some paths disallowed | 12 | robots.txt answered, terms refused 403 | 2 | |
| Naver | Limited access | Every path disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Nike | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Noon | No open API | Could not be read | robots.txt no answer, terms no answer | |||
| OP.GG | Limited access | No disallow rules | 0 | robots.txt answered, terms answered | 1 | |
| Ozon | Partner only | Could not be read | robots.txt answered, terms answered | 2 | ||
| Pexels | Public API | Some paths disallowed | 5 | robots.txt answered, terms refused 403 | 3 | |
| Public API | Every path disallowed | 2 | robots.txt answered, terms answered | 1 | ||
| Product Hunt | Public API | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 2 | |
| Property Finder | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| QuintoAndar | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 1 | |
| Quora | No open API | Every path disallowed | 12 | robots.txt answered, terms refused 403 | 3 | |
| Realtor.com | Partner only | Some paths disallowed | 3 | robots.txt answered, terms answered | 0 | |
| Limited access | Every path disallowed | 0 | robots.txt answered, terms answered | 2 | 2 | |
| Redfin | Limited access | Could not be read | robots.txt refused 403, terms answered | 4 | ||
| Reed | Public API | Some paths disallowed | 4 | robots.txt answered, terms answered | ||
| Sephora | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 3 | ||
| Shein | Partner only | Some paths disallowed | 0 | robots.txt answered, terms answered | 4 | |
| Shopee | Partner only | Could not be read | robots.txt answered, terms answered | 2 | ||
| Shopify stores | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Similarweb | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | |
| SimplyHired | No open API | Could not be read | robots.txt refused 403, terms answered | 0 | ||
| Skool | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 4 | |
| Snapchat | Limited access | Some paths disallowed | 25 | robots.txt answered, terms answered | 3 | |
| SofaScore | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 2 | ||
| SoundCloud | Limited access | Some paths disallowed | 17 | robots.txt answered, terms answered | 1 | |
| Spotify | Limited access | Some paths disallowed | 19 | robots.txt answered, terms answered | 3 | |
| Steam | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| StockTwits | Partner only | Could not be read | robots.txt refused 403, terms refused 403 | 0 | ||
| StockX | Limited access | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 3 | |
| Substack | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | |
| Target | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | ||
| The Home Depot | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 1 | ||
| Threads | Limited access | Could not be read | robots.txt answered, terms refused 400 | |||
| Ticketmaster | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 1 | |
| TikTok | Limited access | Some paths disallowed | 17 | robots.txt answered, terms answered | 4 | |
| TikTok Ads Library | Limited access | Every path disallowed | 0 | robots.txt answered, terms answered | 0 | |
| TikTok Shop | Partner only | Could not be read | robots.txt no answer, terms answered | 3 | ||
| Tokopedia | Partner only | Some paths disallowed | 0 | robots.txt answered, terms no answer | 3 | |
| Totaljobs | No open API | Some paths disallowed | 1 | robots.txt answered, terms refused 403 | 0 | |
| Trading Economics | Public API | No disallow rules | 0 | robots.txt answered, terms answered | 0 | |
| Transfermarkt | No open API | No disallow rules | 0 | robots.txt answered, terms refused 405 | 3 | |
| Trendyol | Partner only | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 0 | |
| Trip.com | Partner only | Some paths disallowed | 0 | robots.txt answered, terms refused 432 | 0 | |
| Tripadvisor | Partner only | Some paths disallowed | 14 | robots.txt answered, terms refused 403 | 6 | |
| Trulia | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 1 | |
| Trustpilot | Limited access | Every path disallowed | 19 | robots.txt answered, terms refused 403 | 2 | |
| TrustRadius | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 0 | ||
| Twitch | Public API | Some paths disallowed | 1 | robots.txt answered, terms answered | 4 | |
| Uber Eats | Partner only | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | |
| Ubersuggest | Limited access | Some paths disallowed | 10 | robots.txt answered, terms refused 403 | 0 | |
| Udemy | Retired | Could not be read | robots.txt refused 403, terms refused 403 | 1 | ||
| Unsplash | Public API | Some paths disallowed | 3 | robots.txt answered, terms refused 401 | 0 | |
| Vimeo | Public API | Some paths disallowed · delay 2 s | 10 | robots.txt answered, terms answered | 3 | |
| Vinted | Limited access | Some paths disallowed | 5 | robots.txt answered, terms answered | 4 | |
| VivaReal | No open API | Could not be read | robots.txt refused 403, terms refused 403 | 3 | ||
| Vivino | No open API | Some paths disallowed | 7 | robots.txt answered, terms answered | 0 | |
| Vrbo | Partner only | Some paths disallowed | 7 | robots.txt answered, terms refused 429 | 0 | |
| Walk Score | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Wallapop | Limited access | Could not be read | robots.txt refused 403, terms refused 403 | 1 | ||
| Walmart | Partner only | Some paths disallowed | 0 | robots.txt answered, terms answered | 3 | |
| Wayback Machine | Public API | Could not be read | robots.txt no answer, terms answered | 7 | ||
| We Work Remotely | Public API | Some paths disallowed | 0 | robots.txt answered, terms refused 403 | 0 | |
| Welcome to the Jungle | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Wellfound | No open API | Some paths disallowed | 0 | robots.txt answered, terms not read | 0 | |
| Wikipedia | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | |
| Wildberries | Limited access | Could not be read | robots.txt refused 498, terms answered | 1 | ||
| Willhaben | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 0 | |
| Workday career sites | Limited access | Could not be read | robots.txt no answer, terms answered | 0 | ||
| X (Twitter) | Public API | Every path disallowed · delay 1 s | 4 | robots.txt answered, terms answered | 1 | 0 |
| Y Combinator | No open API | Some paths disallowed | 0 | robots.txt answered, terms answered | 2 | |
| Yahoo Finance | No open API | Some paths disallowed | 16 | robots.txt answered, terms answered | 2 | |
| Yahoo News | No open API | Some paths disallowed | 17 | robots.txt answered, terms answered | 0 | |
| Yahoo! Auctions Japan | Retired | Could not be read | robots.txt no answer, terms answered | 0 | ||
| Yandex | Limited access | Some paths disallowed | 0 | robots.txt answered, terms answered | 1 | |
| Yelp | Public API | Every path disallowed | 14 | robots.txt answered, terms answered | 0 | |
| YouTube | Public API | Some paths disallowed | 0 | robots.txt answered, terms answered | 4 | |
| Zillow | Partner only | Some paths disallowed | 8 | robots.txt answered, terms answered | 4 | |
| ZipRecruiter | Partner only | Every path disallowed | 11 | robots.txt answered, terms refused 403 | 0 |
Method
The 172 websites are the ones this atlas maps: each has its own page with every route to its data. For each one the study combines facts the site already keeps, each with the date it was read (9 November 2026 to 13 November 2026 for this edition):
- Official API: the website's own developer documentation and pricing pages, each source dated. Public means anyone can sign up and call it; limited means approval, a paid tier or a narrow scope; partner means contracts only.
- robots.txt: fetched from the website (or its www host) with one HTTPS request a week. The rules counted are the ones under
User-agent: *; AI crawlers count when a group names them. - Plain request: the HTTP status of that robots.txt request and of one request for the terms page, with an ordinary browser user-agent string but no browser, cookies or retries. 404 is counted as not found, a timeout as no answer, any other 4xx or 5xx as refused.
- Court cases: a curated list of decided or settled cases about collecting that website's data, each with its source.
- Open-source scrapers: public GitHub repositories whose name carries the website and whose description says they scrape, crawl, download or export it; archived ones are left out.
What this does not measure
Whether a page needs a real browser to render, shows CAPTCHAs, rate-limits, sits behind a login or blocks IP ranges. One plain request answers none of that, and nothing here was run against the websites beyond those two files. A refused request tells you the website screens automated traffic at its front door; an answered one tells you nothing about its search or product pages.
robots.txt is a crawling convention, not a contract, and the terms page is the contract. This is general information, not legal advice.
Download the data
One row per website: official API status and name, both plain-request answers with their HTTP status, the robots.txt rule and rule count, crawl delay, AI crawlers named, court cases, open-source repositories, every date, and the website's page.
Licensed CC BY 4.0: free to reuse with a credit and a link to this page.
Cite this study
Scraper Finder (2026). What it takes to get data from 172 major websites: official APIs, robots.txt rules and plain-request answers. Updated 13 November 2026. https://scraperfinder.com/what-it-takes/
Questions people ask
Does a refused request mean the website blocks scraping?
No. It means one plain HTTPS request, with no browser and no cookies, got an error status back. Many sites answer a browser and refuse a script, others refuse anything from a data-centre address. It is a first signal of how a site treats automated requests, not a test of any tool.
Does robots.txt decide whether collecting data is allowed?
No. robots.txt is a convention for crawlers, and the terms of service are the contract. Each website's page here quotes both with the date they were read. This is general information, not legal advice.
Why are some websites marked as not read?
The request timed out, the connection failed, or the file did not exist. Those websites are counted as not read, never guessed.
How often is it updated?
robots.txt files, terms pages and open-source repositories are read again every week, official API records when their sources change. The date at the top only moves when a number changed.