Development

BrightData

4-tier progressive web scraping that auto-escalates WebFetch to curl to Interceptor to Bright Data proxy for bot detection and CAPTCHAs, with single-URL and multi-page crawl modes, output as markdown

Workflows

Tools

References

Triggers

The Problem

A lot of the web won't let you in. Paywalled content, JavaScript-heavy SPAs that render nothing server-side, sites that fingerprint your headers and return a 403, pages behind CAPTCHAs. A bare WebFetch call works fine on simple public pages and fails silently or noisily on everything else. Without escalation logic, you either get blocked and give up, or you go straight to an expensive proxy service for content you could have fetched in 200ms.

How This Skill Approaches It

Four-tier progressive escalation: start cheap and fast, escalate only when blocked. Tier 1 is WebFetch — built-in, instant, costs nothing. Tier 2 is curl with Chrome-like headers, which clears basic user-agent checks. Tier 3 is agent-browser, a headless Rust CLI daemon that handles JavaScript rendering for dynamic SPAs. Tier 4 is the Bright Data MCP proxy — residential IPs, CAPTCHA solving, advanced bot-detection bypass — which has real usage costs and only runs when the first three tiers fail. FourTierScrape runs this escalation for a single URL. Crawl handles multi-page work: light crawl uses scrape_batch in a loop for up to 50 pages, full crawl hits the Bright Data Crawl API for entire sites. All output lands as markdown.

Not for simple public content (use WebFetch directly), social platform scraping with named actors (use Apify), or real-Chrome bot bypass with logged-in sessions and zero CDP fingerprint (use Interceptor)

In Action

What you say to your DA, and what the BrightData skill actually does.

You say "scrape this page for me — it keeps blocking my requests"

Runs FourTierScrape: tries WebFetch, escalates to curl with Chrome headers, then agent-browser for JS rendering, then Bright Data MCP if all else fails — returns the page content as markdown.
You say "crawl all the pages under the docs section of this site"

Runs Crawl: light crawl mode uses scrape_batch in a loop with link extraction, up to 50 pages; if the site is larger, escalates to the Bright Data Crawl API — returns a site map plus page contents as markdown with crawl stats.

Inside the Skill

The thinking, frameworks, and architecture that distinguish this skill from a generic version of the same task.

The four tiers (tool contract)

Tier	Tool	Wins on	Cost / latency
1	WebFetch	public content, no bot detection	free · ~2-5s
2	curl + Chrome headers	user-agent / basic header checks	free · ~3-7s
3	Interceptor (real Chrome)	JavaScript-rendered / SPA pages	free · ~10-20s
4	Bright Data MCP `mcp__Brightdata__scrape_as_markdown`	CAPTCHA, advanced fingerprinting, residential-IP needs	Bright Data credits · ~5-15s

Playwright is banned across LifeOS — Tier 3 is Interceptor. Skip-ahead: explicit "use Bright Data" → Tier 4; "use browser" → Tier 3; a domain that already failed Tier 1 → start at Tier 2. The exact curl header block, Cloudflare pre-check, and Interceptor commands live in Workflows/FourTierScrape.md.

Workflows

When routing, output: Running the **WorkflowName** workflow in the **BrightData** skill to ACTION...

Workflow	Trigger	File
FourTierScrape	"scrape/fetch/pull/get/retrieve [URL]", "can't access this site", "site is blocking me", "use Bright Data to fetch"	`Workflows/FourTierScrape.md`
Crawl	"crawl this site", "spider this domain", "map this website", "get all pages from", "scrape the whole site", "crawl all pages under /docs"	`Workflows/Crawl.md`

Crawl picks Light Crawl (MCP scrape_batch + link loop, ≤50 pages, ~$0.006/page) for a section, or Full Crawl (Bright Data Crawl API api.brightdata.com/datasets/v3/trigger, $1.50/1K pages) for whole sites.

Gotchas

4-tier escalation: WebFetch → curl → Interceptor → Bright Data proxy. Always start at Tier 1 and escalate only when blocked. Playwright is banned across LifeOS.
Bright Data proxy has usage costs. Don't use Tier 4 for sites accessible via Tier 1-3.
CAPTCHA-solving introduces latency. Allow extra time for Tier 4 responses.
Credentials in ~/.claude/.env — BRIGHTDATA_API_KEY.

Workflows & Routing · 2

Each workflow is one job the skill runs. The trigger phrases route your request to the right one — this is the skill's routing table.

01

FourTierScrape Workflows/FourTierScrape.md

scrape/fetch/pull/get/retrieve [URL], cant access this site, site is blocking me, use Bright Data to fetch
02

Crawl Workflows/Crawl.md

crawl this site, spider this domain, map this website, get all pages from, scrape the whole site, crawl all pages under /docs

How to Invoke

Say any of these to your DA and LifeOS activates the BrightData skill automatically:

"Bright Data"
"scrape URL"
"web scraping"
"bot detection"
"crawl site"
"CAPTCHA"
"can't access"
"site blocking"
"extract page content"
"scrape whole site"
"spider domain"
"convert URL to markdown"
"getting blocked"

Or invoke explicitly:

Skill("BrightData")

Related Skills

References & Credits

The thinkers, books, frameworks, and research this skill is built on. The ideas belong to them — the integration belongs to LifeOS.

Tool

Bright Data Residential proxy network and CAPTCHA-solving layer used at Tier 4 when standard methods are blocked.

Want LifeOS to do this for you?

Install LifeOS on your machine — your DA gets the BrightData skill plus 46 others, all hooked into one Life OS.

Install LifeOS View on GitHub