Domain-Scoped Site Crawler for Markdown Datasets

Crawl a website from one or more start URLs, staying on the same domain, and get back clean Markdown, text, or HTML per page with title, description, and crawl depth. Built for feeding LLM, RAG, or knowledge-base pipelines.

Open in Apify Store$0.99 / 1K on Free

Short answer

Domain-Scoped Site Crawler for Markdown Datasets is an Apify Store Actor that returns 5 published output fields, including title, depth, wordCount, httpStatusCode. It lists $0.99 per 1,000 results on the Free plan (pay per event), $0.49 on the Business plan. It takes 26 input settings; one is required (startUrls). Last updated Oct 17, 2026.

From the public Store listing and published build schema, synced 2026-11-12.
AIAutomationDeveloper tools

What it returns

5 fields from the published dataset schema. Export as JSON, CSV or Excel, or read the dataset through the Apify API.

  • url
  • title
  • depth
  • wordCount
  • httpStatusCode

Input

SettingTypeRequiredDefaultWhat it does
startUrls array Yes – One or more URLs to start crawling from. The crawler follows links found on these pages and stays on the domain of the first start URL.
maxCrawlDepth integer No 10 How many links deep to follow from the start URLs. 0 means only the start URLs are crawled.
maxCrawlPages integer No 20 Maximum number of pages (and files) to store. The crawl stops when this limit is reached.
useSitemaps boolean No false Look for sitemaps (from robots.txt and /sitemap.xml) and enqueue the URLs they list. Finds pages that are not linked from the start URLs.
useLlmsTxt boolean No false Look for /llms.txt at the start URL domains and enqueue the files listed there (common on documentation sites).
respectRobotsTxtFile boolean No true Skip discovered pages that the site's robots.txt disallows. Start URLs are always crawled.
crawlOnlyPagesChangedSince string No – Optional date (YYYY-MM-DD or full ISO date-time). Crawls only pages that may have changed since then and outputs a delta. Leave empty to crawl everything.
includeUrlGlobs array No [] Only crawl discovered URLs matching these glob patterns (e.g. https://example.com/docs/*). Leave empty to allow everything on the domain.
excludeUrlGlobs array No [] Skip discovered URLs matching any of these glob patterns (e.g. *login*, *.pdf).
includeUrlRegexps array No [] Only crawl discovered URLs matching at least one of these regular expressions. Combined with glob patterns (both must pass when both are set).
Show the other 16 settings
SettingTypeRequiredDefaultWhat it does
excludeUrlRegexps array No [] Skip discovered URLs matching any of these regular expressions.
keepUrlFragments boolean No false Treat URLs that differ only in #fragment as different pages. Enable for single-page sites that use fragments as routes.
ignoreCanonicalUrl boolean No false When disabled, pages whose canonical URL points to a page that was already crawled are skipped as duplicates.
outputFormat string No markdown How page content is returned. Markdown keeps headings, lists, tables, and links — best for LLMs and RAG.
htmlTransformer string No auto How the main content is isolated from navigation, footers, ads, and cookie banners.
includeImages boolean No false Add image references to the markdown field. Image URLs are always collected in the imageUrls field.
chunkMarkdown boolean No false Emit a chunks array per page — heading-aware Markdown pieces ready for embeddings. Code blocks and tables are never split, each piece carries its heading path and character count, and consecutive pieces overlap so context is kept at the boundaries.
chunkMaxChars integer No 2000 Target size of each chunk when chunkMarkdown is on. Code blocks and tables stay whole even when larger than this.
chunkOverlapChars integer No 200 Approximate overlap between consecutive chunks when chunkMarkdown is on, so context is not lost at chunk boundaries.
keepElementsCssSelector string No – Keep only the elements matching this CSS selector (e.g. article, main .content) before extraction. Leave empty to use the whole page.
removeElementsCssSelector string No – Remove elements matching this CSS selector (e.g. .sidebar, .ad, nav) before extraction.
parsePdfText boolean No true When a crawled URL serves a PDF, extract its text into the text field as a file record instead of skipping it.
clickElementsCssSelector string No – Click elements matching this CSS selector (e.g. [aria-expanded="false"]) before extraction to expand collapsed content.
takeScreenshots boolean No false Render every stored page in the browser and capture a full-page JPEG (quality 80) of it, putting its public URL in the screenshot field. Screenshots are held in memory until pushed and add roughly 0.3-1 MB and a fraction of a second per page; screenshot capture always runs after the page is rendered
customHttpHeaders array No [] Extra request headers for all engines and discovery requests, as a list of {"key", "value"} pairs (e.g. an Authorization token or a specific User-Agent for sites that ask for one). Transport/framing and cookie headers (Host, Cookie, Content-Length, Connection, ...) are rejected so the actor stays co
storeSkippedUrls boolean No false Save a SKIPPED_URLS record to the key-value store listing skipped URLs grouped by reason (robots, blocked, non-HTML, errors, duplicates).

Example input

Built from the input form's published example values and defaults.

{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "maxCrawlDepth": 1,
  "maxCrawlPages": 3,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": true,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "includeUrlRegexps": [],
  "excludeUrlRegexps": [],
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "outputFormat": "markdown",
  "htmlTransformer": "auto",
  "includeImages": false,
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "parsePdfText": true,
  "takeScreenshots": false,
  "customHttpHeaders": [],
  "storeSkippedUrls": false
}

Run it from code or an AI agent

The same run through the Apify API, with the example input. You need an Apify API token; the run is billed to your account at the Store price.

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run_input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ],
    "maxCrawlDepth": 1,
    "maxCrawlPages": 3,
    "useSitemaps": False,
    "useLlmsTxt": False,
    "respectRobotsTxtFile": True,
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "includeUrlRegexps": [],
    "excludeUrlRegexps": [],
    "keepUrlFragments": False,
    "ignoreCanonicalUrl": False,
    "outputFormat": "markdown",
    "htmlTransformer": "auto",
    "includeImages": False,
    "chunkMarkdown": False,
    "chunkMaxChars": 2000,
    "chunkOverlapChars": 200,
    "parsePdfText": True,
    "takeScreenshots": False,
    "customHttpHeaders": [],
    "storeSkippedUrls": False
}

run = client.actor("vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets").call(run_input=run_input)
if run is None:
    raise RuntimeError("The Actor run was not found.")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

pip install apify-client

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ],
    "maxCrawlDepth": 1,
    "maxCrawlPages": 3,
    "useSitemaps": false,
    "useLlmsTxt": false,
    "respectRobotsTxtFile": true,
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "includeUrlRegexps": [],
    "excludeUrlRegexps": [],
    "keepUrlFragments": false,
    "ignoreCanonicalUrl": false,
    "outputFormat": "markdown",
    "htmlTransformer": "auto",
    "includeImages": false,
    "chunkMarkdown": false,
    "chunkMaxChars": 2000,
    "chunkOverlapChars": 200,
    "parsePdfText": true,
    "takeScreenshots": false,
    "customHttpHeaders": [],
    "storeSkippedUrls": false
};

const run = await client.actor("vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets").call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

npm install apify-client

cURL

curl -X POST "https://api.apify.com/v2/actors/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/run-sync-get-dataset-items"   -H "Authorization: Bearer $APIFY_TOKEN"   -H "Content-Type: application/json"   -d '{"startUrls":[{"url":"https://docs.apify.com"}],"maxCrawlDepth":1,"maxCrawlPages":3,"useSitemaps":false,"useLlmsTxt":false,"respectRobotsTxtFile":true,"includeUrlGlobs":[],"excludeUrlGlobs":[],"includeUrlRegexps":[],"excludeUrlRegexps":[],"keepUrlFragments":false,"ignoreCanonicalUrl":false,"outputFormat":"markdown","htmlTransformer":"auto","includeImages":false,"chunkMarkdown":false,"chunkMaxChars":2000,"chunkOverlapChars":200,"parsePdfText":true,"takeScreenshots":false,"customHttpHeaders":[],"storeSkippedUrls":false}'

Waits for the run and returns the dataset items (runs over 300 seconds return HTTP 408; start an async run instead).

MCP (AI agents)

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Adds this Actor as a tool in an MCP client that takes a remote server URL (Cursor, VS Code, Claude Code) through Apify's hosted MCP server. OAuth sign-in also works without the header. Clients that only start local servers can wrap the URL with npx mcp-remote.

What a run costs

The result rate falls on higher Apify plans. Each figure is the published per-result rate times the number of results.

Apify planPer 1,00010,000 results100,000 results
Free plan$0.99$9.90$99
Starter plan$0.82$8.20$82
Scale plan$0.66$6.60$66
Business plan$0.49$4.90$49
  • Every run also has a start event, charged per GB of memory (minimum one): $0.10 on Free, $0.03 on Business. At its default memory of 4 GB a run starts at $0.40 on Free. Fewer, larger runs keep this small.
  • The Free plan includes $5 of monthly usage (Apify pricing, checked 2026-11-11); at the Free rate that covers at most about 5,050 results, less start events and platform usage.
  • Worked example: 10,000 results in one run on the Free plan cost $10.30 ($9.90 for results + $0.40 start), plus Apify platform usage (compute), billed separately by Apify.
  • Pay per event. Platform usage (compute) is billed by Apify on top of the event prices. Compare other tools in the cost estimator.

What people use it for

  • crawl website pages
  • extract page content

Search & URL Markdown Extractor for LLMs

Turn a search query or a list of URLs into clean Markdown, plain text, or HTML pages, with filters for domain, keyword, and minimum content…

13 output fields $0.99 per 1K

Ahrefs Domain Rating, Backlinks & Traffic Lookup

Look up a domain's public Ahrefs Domain Rating, linking-website count, global rank, and estimated organic traffic, with a mode to fetch just…

6 output fields $0.99 per 1K

Amazon Brand Store Page & Merchandising Crawler

Crawl an Amazon Brand Store's navigation, creative modules (images, video, CTAs), and every product link found on its pages. Resolves a brand's…

11 output fields $0.99 per 1K

Tor Hidden Service Crawler & Contact Extractor

Crawl Tor .onion sites by URL or keyword search and pull emails, phone numbers, crypto wallet addresses, social handles, and exposed API keys from…

6 output fields $0.99 per 1K

Website Contact Page Email & Phone Crawler

Crawl a website's contact, about and team pages and collect its public emails, phone numbers, and social media profile links in one normalized…

$0.99 per 1K

SimilarWeb Traffic & AI Chatbot Analytics Scraper

Look up website traffic estimates, rankings, traffic-source breakdown, top keywords, and AI-chatbot traffic share for any list of domains, with no…

9 output fields $0.99 per 1K

Questions people ask

How much does Domain-Scoped Site Crawler for Markdown Datasets cost?

The published result rate depends on your Apify plan: $0.99 per 1,000 on Free, $0.82 per 1,000 on Starter, $0.66 per 1,000 on Scale, $0.49 per 1,000 on Business. So 10,000 results cost about $9.90 on the Free plan and $4.90 on Business, plus a start fee of $0.10 per run per GB of memory on Free. Platform usage is billed separately.

What data does Domain-Scoped Site Crawler for Markdown Datasets return?

Its published output schema lists 5 fields: url, title, depth, wordCount, httpStatusCode. A listed field is not guaranteed to be filled in every row.

What input does Domain-Scoped Site Crawler for Markdown Datasets need?

Required: startUrls. Optional settings include maxCrawlDepth, maxCrawlPages, useSitemaps, useLlmsTxt, respectRobotsTxtFile, crawlOnlyPagesChangedSince.

How do I run Domain-Scoped Site Crawler for Markdown Datasets from Python or the API?

Use the Apify API client: in Python, ApifyClient(token).actor("vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets").call(run_input=...), then read the run's default dataset (run.default_dataset_id); in JavaScript the same call is client.actor("vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets").call(input). Without a client, POST the input JSON to https://api.apify.com/v2/actors/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/run-sync-get-dataset-items with your API token. Ready-to-paste code with the example input is on this page.

Sources

Actor updated 2026-10-17 Data synced 2026-11-12
Cite this page

Scraper Finder. "Domain-Scoped Site Crawler for Markdown Datasets." https://scraperfinder.com/catalogue/domain-scoped-site-crawler-for-markdown-datasets/ (data checked 2026-11-12).

Run Domain-Scoped Site Crawler for Markdown Datasets

$0.99 per 1,000 results. Paste the example input above, set your target query, then export the dataset.

Open in Apify Store