4-tier page fetching that moves from plain curl to curl with declared headers to a real browser through Interceptor to Bright Data's managed fetch product, single URL or multi-page crawl, output as markdown, within each target site's terms
A lot of the web won't let you in. Paywalled content, JavaScript-heavy SPAs that render nothing server-side, sites that fingerprint your headers and return a 403, pages behind CAPTCHAs. A bare WebFetch call works fine on simple public pages and fails silently or noisily on everything else. Without escalation logic, you either get blocked and give up, or you go straight to an expensive proxy service for content you could have fetched in 200ms.
Four-tier progressive escalation: start cheap and fast, escalate only when blocked. Tier 1 is WebFetch — built-in, instant, costs nothing. Tier 2 is curl with Chrome-like headers, which clears basic user-agent checks. Tier 3 is agent-browser, a headless Rust CLI daemon that handles JavaScript rendering for dynamic SPAs. Tier 4 is the Bright Data MCP proxy — residential IPs, CAPTCHA solving, advanced bot-detection bypass — which has real usage costs and only runs when the first three tiers fail. FourTierScrape runs this escalation for a single URL. Crawl handles multi-page work: light crawl uses scrape_batch in a loop for up to 50 pages, full crawl hits the Bright Data Crawl API for entire sites. All output lands as markdown.
What you say to your DA, and what the BrightData skill actually does.
The thinking, frameworks, and architecture that distinguish this skill from a generic version of the same task.
| Intent | Trigger | File |
|---|---|---|
| FourTierScrape | "scrape/fetch/pull/get/retrieve [URL]", "fetch this page as markdown", "this page needs rendering", "use Bright Data to fetch" | Intents/FourTierScrape.md |
| Crawl | "crawl this site", "spider this domain", "map this website", "get all pages from", "scrape the whole site", "crawl all pages under /docs" | Intents/Crawl.md |
| File | Read when |
|---|---|
Setup.md |
first-time setup: the free tiers, the paid Bright Data pieces (MCP server, API key, crawl dataset id), and a proof |
| Tier | Tool | Wins on | Cost / latency |
|---|---|---|---|
| 1 | plain curl -sL, raw HTML converted to markdown locally |
public content | free · ~1-3s |
| 2 | curl + declared user agent and Accept headers | servers that need content negotiation | free · ~3-7s |
| 3 | Interceptor (real Chrome) | JavaScript-rendered / SPA pages | free · ~10-20s |
| 4 | Bright Data MCP mcp__Brightdata__scrape_as_markdown |
managed rendering, region-specific fetches | Bright Data credits · ~5-15s |
Every tier returns the page itself. WebFetch is never a tier: it hands back a small model's summary of the page, not the page.
Skip-ahead: explicit "use Bright Data" → Tier 4; "use browser" → Tier 3; a domain that already failed Tier 1 for a non-refusal reason (empty body, timeout) → start at Tier 2. A CAPTCHA, block page or login wall at any tier ends the retrieval with a report. The exact curl header block, Cloudflare pre-check, and Interceptor commands live in Intents/FourTierScrape.md.
User customizations live at <configRoot>/LIFEOS/USER/CUSTOMIZATIONS/SKILLS/BrightData/ (configRoot is $CLAUDE_CONFIG_DIR when set, else ~/.claude). If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there; they override default behavior. If it does not exist, proceed with skill defaults. First-time setup (accounts, keys, config, a proof it works) is in Setup.md.
The paid paths need pieces this skill does not install. Tier 4 needs the Bright Data MCP server, which supplies mcp__Brightdata__scrape_as_markdown and scrape_batch. Full Crawl needs BRIGHT_DATA_API_KEY and BRIGHT_DATA_CRAWL_DATASET_ID (the Crawl API dataset id from the Bright Data dashboard) in the environment. Before the first paid call, confirm the MCP tools are present in the session and that bun LIFEOS/TOOLS/Secrets.ts has BRIGHT_DATA_API_KEY BRIGHT_DATA_CRAWL_DATASET_ID prints yes for each key the call needs. When a piece is missing, stop at the last free tier and tell the user which piece is not configured; never report a Tier 4 or Full Crawl result that did not run.
~/.claude/.env — BRIGHT_DATA_API_KEY (the name Intents/Crawl.md commands read). Check as Data & Config says instead of assuming the paid tiers work./api/, depth 3, at the default limit of 20 pages, which needs no confirmation (more than 20 does). Each page climbs the tiers on its own, and any page that needed Tier 4 is named in the stats with its cost.Each workflow is one job the skill runs. The trigger phrases route your request to the right one — this is the skill's routing table.
Intents/FourTierScrape.md scrape/fetch/pull/get/retrieve [URL], fetch this page as markdown, this page needs rendering, use Bright Data to fetch
Intents/Crawl.md crawl this site, spider this domain, map this website, get all pages from, scrape the whole site, crawl all pages under /docs
Deterministic executables the workflows call — the code that does the real work, not prompt scaffolding.
MarkdownPreCheck.tsSay any of these to your DA and LifeOS activates the BrightData skill automatically:
Or invoke explicitly:
Skill("BrightData")Auxiliary files the skill loads at runtime — frameworks, guides, configs.
The thinkers, books, frameworks, and research this skill is built on. The ideas belong to them — the integration belongs to LifeOS.
Install LifeOS on your machine — your DA gets the BrightData skill plus 96 others, all hooked into one Life OS.