Website crawler to Markdown in n8n
Crawl a website from one or more start URLs, staying on the same domain, and get back clean Markdown, text, or HTML per page with title, description, and crawl depth. Built for feeding LLM, RAG, or knowledge-base pipelines.
What this workflow does
Import it and n8n runs the Domain-Scoped Site Crawler for Markdown Datasets Actor on Apify every day at 08:00 with the starter input below. Each row comes back with url, title, depth, words and 1 more; repeats with the same url are dropped. A Google Sheets append is included and stays off until you add your sheet. The only credential it needs is your Apify API token.
Read from the workflow file, built for Domain-Scoped Site Crawler for Markdown Datasets version 1.1.1. Updated 2026-11-14.How the rows travel
6 nodes, in the order n8n runs them. The yellow stop is the call to the Apify Actor; the dashed stop is optional.
- Daily trigger Schedule trigger
- Configure input Code
- Run Apify Actor HTTP Request
- Normalize + dedupe Code
- Rows ready No operation
- Google Sheets (optional) Google Sheets, off until you switch it on
Input
The Configure input node holds the starter input. Change it there; every setting the Actor accepts is listed below.
{
"startUrls": [
{
"url": "https://docs.apify.com"
}
],
"maxCrawlDepth": 1,
"maxCrawlPages": 3
}| Setting | Type | Required | Default | What it does |
|---|---|---|---|---|
| startUrls | array | Yes | – | One or more URLs to start crawling from. The crawler follows links found on these pages and stays on the domain of the first start URL. |
| maxCrawlDepth | integer | No | 10 | How many links deep to follow from the start URLs. 0 means only the start URLs are crawled. |
| maxCrawlPages | integer | No | 20 | Maximum number of pages (and files) to store. The crawl stops when this limit is reached. |
| useSitemaps | boolean | No | false | Look for sitemaps (from robots.txt and /sitemap.xml) and enqueue the URLs they list. Finds pages that are not linked from the start URLs. |
| useLlmsTxt | boolean | No | false | Look for /llms.txt at the start URL domains and enqueue the files listed there (common on documentation sites). |
| respectRobotsTxtFile | boolean | No | true | Skip discovered pages that the site's robots.txt disallows. Start URLs are always crawled. |
| crawlOnlyPagesChangedSince | string | No | – | Optional date (YYYY-MM-DD or full ISO date-time). Crawls only pages that may have changed since then and outputs a delta. Leave empty to crawl everything. |
| includeUrlGlobs | array | No | – | Only crawl discovered URLs matching these glob patterns (e.g. https://example.com/docs/*). Leave empty to allow everything on the domain. |
| excludeUrlGlobs | array | No | – | Skip discovered URLs matching any of these glob patterns (e.g. *login*, *.pdf). |
| includeUrlRegexps | array | No | – | Only crawl discovered URLs matching at least one of these regular expressions. Combined with glob patterns (both must pass when both are set). |
| excludeUrlRegexps | array | No | – | Skip discovered URLs matching any of these regular expressions. |
| keepUrlFragments | boolean | No | false | Treat URLs that differ only in #fragment as different pages. Enable for single-page sites that use fragments as routes. |
| ignoreCanonicalUrl | boolean | No | false | When disabled, pages whose canonical URL points to a page that was already crawled are skipped as duplicates. |
| outputFormat | string | No | markdown | How page content is returned. Markdown keeps headings, lists, tables, and links — best for LLMs and RAG. One of: markdown, text, html. |
| htmlTransformer | string | No | auto | How the main content is isolated from navigation, footers, ads, and cookie banners. One of: auto, trafilatura, readability, basic, none. |
| includeImages | boolean | No | false | Add image references to the markdown field. Image URLs are always collected in the imageUrls field. |
| chunkMarkdown | boolean | No | false | Emit a chunks array per page — heading-aware Markdown pieces ready for embeddings. Code blocks and tables are never split, each piece carries its heading path and character count… |
| chunkMaxChars | integer | No | 2000 | Target size of each chunk when chunkMarkdown is on. Code blocks and tables stay whole even when larger than this. |
| chunkOverlapChars | integer | No | 200 | Approximate overlap between consecutive chunks when chunkMarkdown is on, so context is not lost at chunk boundaries. |
| keepElementsCssSelector | string | No | – | Keep only the elements matching this CSS selector (e.g. article, main .content) before extraction. Leave empty to use the whole page. |
| removeElementsCssSelector | string | No | – | Remove elements matching this CSS selector (e.g. .sidebar, .ad, nav) before extraction. |
| parsePdfText | boolean | No | true | When a crawled URL serves a PDF, extract its text into the text field as a file record instead of skipping it. |
| clickElementsCssSelector | string | No | – | Click elements matching this CSS selector (e.g. [aria-expanded="false"]) before extraction to expand collapsed content. |
| takeScreenshots | boolean | No | false | Render every stored page in the browser and capture a full-page JPEG (quality 80) of it, putting its public URL in the screenshot field. |
| customHttpHeaders | array | No | – | Extra request headers for all engines and discovery requests, as a list of {"key", "value"} pairs (e.g. |
| storeSkippedUrls | boolean | No | false | Save a SKIPPED_URLS record to the key-value store listing skipped URLs grouped by reason (robots, blocked, non-HTML, errors, duplicates). |
What each row holds
Every field the Actor returns stays in the row (5 in its dataset schema). Normalize + dedupe trims text, drops empty fields, removes repeats with the same url and adds _source and _scrapedAt. The main fields:
| Field | What it is |
|---|---|
| url | URL |
| title | Title |
| depth | Depth |
| wordCount | Words |
| httpStatusCode | Status |
What the first run costs
Each run starts at $0.40 on the Free plan (4 GB of memory), then $0.99 per 1,000 results, plus platform usage.
The Actor lists $0.99 per 1,000 results; its catalogue page breaks the price down by Apify plan. The Apify Free plan includes $5 of usage a month.
The Actor behind it
Domain-Scoped Site Crawler for Markdown Datasets: Crawl a website from one or more start URLs, staying on the same domain, and get back clean Markdown, text, or HTML per page with title, description, and crawl depth. Built for feeding LLM, RAG, or knowledge-base pipelines.
The same job from an AI agent
The matching agent skill gives Claude Code, Codex, Cursor and other agents the same Actor, input and fields. Install it in one line:
npx skills add https://scraperfinder.com --skill domain-scoped-site-crawler-for-markdown-datasets The website crawler to Markdown skill, with manual install and run steps
Questions people ask
How do I import this workflow into n8n?
Copy https://scraperfinder.com/n8n/domain-scoped-site-crawler-for-markdown-datasets.json. In n8n, open a new workflow, click the three-dot menu, choose Import from URL and paste it. Or download the JSON file and use Import from File; both give the same 6 nodes.
Do I need an Apify account?
Yes. The workflow calls the Domain-Scoped Site Crawler for Markdown Datasets Actor through the Apify API with your own token, and each run is billed to your Apify account. The Free plan includes $5 of usage a month.
What does a run cost?
Each run starts at $0.40 on the Free plan (4 GB of memory), then $0.99 per 1,000 results, plus platform usage. The Actor's Store page shows the current price.
Can I change what it collects?
Yes: edit the input object in the Configure input node. The Actor takes 26 settings; the table on this page lists each one with its default.
Does it work without Google Sheets?
Yes. The rows wait in the Rows ready node, so you can connect any other node there: a database, Airtable, a webhook or an email. The Google Sheets node is only one option.