403 Forbidden with cf-mitigated: challenge: Cloudflare blocks

A 403 from a site behind Cloudflare is either a challenge page or a block rule. How to tell which one you got, what Cloudflare documents as the causes, and how clearance works.

HTTP 403 Forbidden, cf-mitigated: challenge

Short answer

A cf-mitigated header set to challenge identifies a Cloudflare challenge page returned in place of the requested content. A security rule with a Block action is a different cause of denial, so inspect the response before deciding how to proceed.

Diagnosis checked 2026-11-12; sources below.

Why it happens

Cloudflare answered instead of the website: a cf-mitigated header with the value challenge marks a challenge page, and error 1020 is a 403 sent when a security rule with a Block action matched. Source: Cloudflare: detect a challenge response.

Do this now

  1. For public Indeed data at volume, run a hosted route instead of patching your own requests: Indeed Job Search Scraper with Salary & Benefits Data. The request below runs it with its published example input through the Apify API (your token, billed at the Store price).
  2. Compare every route, with fields and prices, on the Indeed page.

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run_input = {
    "searchQueries": [
        "web developer in San Francisco"
    ],
    "country": "US",
    "location": "San Francisco",
    "radius": 25,
    "radiusUnit": "miles",
    "sort": "relevance",
    "easyApplyOnly": False,
    "maxItems": 3,
    "includeDescriptionHtml": False,
    "includeCompanyRatings": False,
    "includeContacts": False,
    "includeCompanyReviews": False,
    "includeCompanySalaries": False,
    "expandBeyondCeiling": False
}

run = client.actor("vexaloom-labs/indeed-job-search-scraper-with-salary-and-benefits").call(run_input=run_input)
if run is None:
    raise RuntimeError("The Actor run was not found.")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

pip install apify-client

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });
const input = {
    "searchQueries": [
        "web developer in San Francisco"
    ],
    "country": "US",
    "location": "San Francisco",
    "radius": 25,
    "radiusUnit": "miles",
    "sort": "relevance",
    "easyApplyOnly": false,
    "maxItems": 3,
    "includeDescriptionHtml": false,
    "includeCompanyRatings": false,
    "includeContacts": false,
    "includeCompanyReviews": false,
    "includeCompanySalaries": false,
    "expandBeyondCeiling": false
};

const run = await client.actor("vexaloom-labs/indeed-job-search-scraper-with-salary-and-benefits").call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

npm install apify-client

cURL

curl -X POST "https://api.apify.com/v2/actors/vexaloom-labs~indeed-job-search-scraper-with-salary-and-benefits/run-sync-get-dataset-items"   -H "Authorization: Bearer $APIFY_TOKEN"   -H "Content-Type: application/json"   -d '{"searchQueries":["web developer in San Francisco"],"country":"US","location":"San Francisco","radius":25,"radiusUnit":"miles","sort":"relevance","easyApplyOnly":false,"maxItems":3,"includeDescriptionHtml":false,"includeCompanyRatings":false,"includeContacts":false,"includeCompanyReviews":false,"includeCompanySalaries":false,"expandBeyondCeiling":false}'

Waits for the run and returns the dataset items (runs over 300 seconds return HTTP 408; start an async run instead).

MCP (AI agents)

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=vexaloom-labs/indeed-job-search-scraper-with-salary-and-benefits",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Adds this Actor as a tool in an MCP client that takes a remote server URL (Cursor, VS Code, Claude Code) through Apify's hosted MCP server. OAuth sign-in also works without the header. Clients that only start local servers can wrap the URL with npx mcp-remote.

Open Indeed Job Search Scraper with Salary & Benefits Data in Apify Store

Routes that work for Indeed data

The official route first, then the catalogue Actors that fit this problem, by their published fields. Store figures are the publisher's public numbers.

RouteCore fieldsPrice / 1KSuccess, 30 days
Partner onlyIndeed partner APIsnot comparedbelowOfficial docs
Indeed Job Search Scraper with Salary & Benefits Data6/7$0.99100% of 26 runsOpen in Apify Store
Indeed Global Job Scraper with Company Salary Reports6/7$0.99100% of 28 runsOpen in Apify Store

Official price: Partner terms; no public self-serve rate card. Compare all 11 Indeed routes.

Routes that work for Product Hunt data

The official route first, then the catalogue Actors that fit this problem, by their published fields. Store figures are the publisher's public numbers.

RouteCore fieldsPrice / 1KSuccess, 30 days
Public APIProduct Hunt API v2not comparedbelowOfficial docs
Product Hunt Launch & Topic Data Exporter3/6$0.9919 runs, too few to rateOpen in Apify Store

Official price: No per-call fee is published for standard API use. Compare all 9 Product Hunt routes.

The facts behind the error

FactValue
Challenge markerA cf-mitigated header with the value challenge marks a Cloudflare challenge page; challenge is its only valid value
Challenge bodyChallenge responses always come back as text/html, whatever resource was requested
Block rulesError 1020 Access denied is an HTTP 403 sent when a security rule with a Block action matched
Other 403 causesWAF managed or custom rules, Security Level, DDoS protection, Browser Integrity Check and most 1xxx errors
ClearancePassing a challenge sets a cf_clearance cookie, valid 30 minutes by default (15-45 minutes recommended)
Cloudflare vs originA Cloudflare-generated 403 carries Cloudflare branding; an origin 403 does not
Verified botsCloudflare verifies bots that identify themselves and respect robots.txt

Identify the challenge response

Check for cf-mitigated with the value challenge and inspect the content type. Cloudflare challenge responses use text/html regardless of the resource requested, so changing selectors or JSON parsing will not recover the intended content.

Separate a challenge from a block

Error 1020 means a security rule with a Block action matched. Other Cloudflare denials can involve firewall rules, Security Level, DDoS protection or Browser Integrity Check; use Cloudflare branding to distinguish a Cloudflare-generated denial from an origin response.

Understand clearance

Passing a challenge sets a cf_clearance cookie with a limited lifetime. Account for that expiry when handling an authorized browser session, and distinguish clearance handling from a security rule that explicitly blocks the request.

When to stop fighting the error

Parser changes are pointless while Cloudflare is returning a challenge page or an explicit block. If an authorized browser session cannot support the collection job, ask the site owner about an API, export or approved automated access route; a blocking rule needs an access decision before collection can continue.

Questions people ask

Why did a JSON request receive HTML?

Cloudflare challenge responses always use text/html, whatever resource was requested. Check the challenge header before passing the body to a JSON parser.

Does every Forbidden response mean there is a challenge?

Forbidden responses can also come from blocking rules or the origin server. The cf-mitigated header with the value challenge identifies a Cloudflare challenge page.

Does passing a challenge give permanent access?

Challenge passage creates a cf_clearance cookie with an expiry. Plan for its limited lifetime when maintaining an authorized session.

How can I tell whether Cloudflare or the website denied access?

Cloudflare-generated Forbidden responses carry Cloudflare branding, while origin responses do not. Use that distinction to direct troubleshooting toward Cloudflare security settings or the origin.

Sources

Every dated claim on this page comes from the public sources below. Check the source before you make a business decision on it.

Content updated 2026-11-12Data checked 2026-11-12

Get Indeed data with a catalogue Actor

Indeed Job Search Scraper with Salary & Benefits Data returns 6 of 7 core Indeed fields at $0.99 per 1,000 results. Compare all 11 Indeed routes, or see Product Hunt, Quora, G2 and Glassdoor.

Open in Apify Store