Crawlee

Crawlee is the open-source web-scraping and browser-automation framework built by Apify, available for Node.js (TypeScript) and Python, with first-class support for Cheerio, Puppeteer and Playwright behind a single, autoscaling crawler runtime. We rate it 90/100 — the most production-ready open-source scraping toolkit shipping today, and the right default for any team building serious crawlers in 2026.

What is Crawlee?

Crawlee is the in-house framework that powers Apify's own commercial scraping platform, open-sourced under Apache 2.0 in August 2022 as a successor to the older Apify SDK. It abstracts the messy parts of crawling — request queues, retries, browser fingerprints, proxy rotation, session pools, autoscaling, error recovery — behind one consistent API, and it lets you swap between an HTTP-only crawler (Cheerio / BeautifulSoup) and a full browser crawler (Puppeteer or Playwright) without rewriting your scraping logic.

The library is maintained by Prague-based Apify, the same team that runs the Apify cloud platform and Actor marketplace. Crawlee for JavaScript currently sits at 23,000+ GitHub stars, 1,340 forks and shipped v3.16.0 in February 2026. Crawlee for Python — launched in July 2024 — has crossed 8,800 stars and pushed v1.6.3 on April 27, 2026, putting both ports on weekly to fortnightly release cadences.

Key Features of Crawlee

• Three crawler classes, one interface: CheerioCrawler, PuppeteerCrawler and PlaywrightCrawler share the same router, queue and storage APIs — switch from a fast HTML-only crawl to a full headless browser by changing a single import.
• Realistic browser fingerprints out of the box: Crawlee ships fingerprint-suite, which generates human-like TLS, header and Canvas/WebGL fingerprints from real browsers, so headless Playwright runs aren't trivially blocked by the major anti-bot vendors.
• Smart proxy rotation: Built-in ProxyConfiguration rotates proxies based on success rate and per-session stickiness, with first-class hooks for Apify Proxy, Bright Data, Smartproxy and any custom HTTP/SOCKS5 list.
• Persistent request queue: A SQLite-backed queue survives process crashes — restart your crawler and it resumes from the last unprocessed URL. The same storage abstraction also persists scraped datasets and key-value stores.
• Adaptive autoscaling: The AutoscaledPool watches CPU, memory and event-loop lag and ramps concurrency up or down automatically — no manual maxConcurrency guessing.
• Adaptive Playwright crawler (3.10+): The newer AdaptivePlaywrightCrawler tries each URL with cheap HTTP first and only falls back to a full browser when the page actually needs JavaScript — cutting compute by 5–10× on mixed sites.
• Crawlee for Python with full feature parity: Released July 2024, it gives Python teams the same queue, proxy and fingerprinting stack on top of HTTPX, BeautifulSoup, Parsel and Playwright.
• One-command CLI scaffolds: npx crawlee create spins up a Cheerio, Playwright or TypeScript starter project with Docker, ESLint and TypeScript wired up — under 30 seconds to first run.

What Users Say About Crawlee

Sentiment is overwhelmingly positive among production scraping teams. On Hacker News, the recurring praise is that Crawlee is the only popular scraper that "feels like a real framework instead of a stitched-together tutorial" — the queue, retry and storage primitives are exactly what people end up reinventing if they roll their own. On Reddit's r/webscraping, the consensus across multiple 2025 and 2026 threads is that Crawlee + Playwright is the default recommendation for anyone past the toy stage, with Scrapy being the only serious Python alternative for veterans who want maximum control.

The honest complaints are mostly about scope and learning curve. The TypeScript types are dense, the docs assume you already know why you'd want a request queue, and the Python port still has a smaller plugin ecosystem than the JS port. A handful of users report that Crawlee's default fingerprints get caught by Cloudflare's stricter Turnstile rules — you still need a residential proxy and sometimes a stealth plugin for the hardest targets. None of those are dealbreakers, but they're worth knowing before you commit.

Crawlee Pricing

Crawlee the library is and will remain free and open-source under Apache 2.0 — you can run it on your laptop, on a $5 VPS or on Kubernetes without paying anyone. Pricing only enters the picture if you decide to host crawlers on Apify's managed cloud or use Apify Proxy. The Apify platform tiers as of 2026 are below.

Plan	Price	Key Limits
Free (Apify cloud)	$0/month	$5 in monthly platform credits, 7-day data retention. Plenty for prototyping.
Starter	$29/month	$39 platform credits, 14-day retention, email support.
Scale	$199/month	$249 platform credits, 30-day retention, priority chat support.
Business	$999/month	$1,249 platform credits, premium support, account manager.
Self-hosted Crawlee	$0	Free forever, Apache 2.0, run anywhere with no token-counting.
Enterprise	Custom	SSO, SOC 2, contractual SLA, dedicated infra.

Who Should Use Crawlee?

Best for: Engineering teams building production scrapers, RAG ingestion pipelines, price-monitoring crawlers or LLM training datasets — anyone who needs proxy rotation, fingerprinting and recoverable state but doesn't want to reinvent the queue. Solo developers also love it because npx crawlee create gets you to a running, dockerised crawler faster than rolling your own Playwright script.

Not ideal for: One-off five-minute scrapes where a 50-line Python script and requests would do the job, or for non-developers who want a no-code visual builder — for that, Apify Actors or Octoparse will fit better.

Pros and Cons

Pros:

• Genuinely production-grade primitives (queue, retries, autoscaling) instead of a thin Playwright wrapper.
• Free, Apache 2.0, no rug-pull risk — you can fork it or deploy on your own infra forever.
• Excellent default fingerprints solve a problem most teams hit at hour two of their first real scraping project.
• One framework, two languages — rare, and a real productivity win for polyglot teams.
• Active maintenance: weekly to fortnightly releases, responsive Discord, public roadmap.

Cons:

• Steep concept curve — routers, request queues and session pools all need to click before it feels easy.
• Default fingerprints don't beat Cloudflare Turnstile or DataDome on the hardest sites without a residential proxy.
• Python port still has a smaller plugin/extension ecosystem than the JS port.
• The Apify cloud upsell is gentle but ever-present in the docs — some users find it noisy.

Alternatives to Crawlee

Scrapy is the long-standing Python alternative — battle-tested, plugin-rich, but built around a 2010-era async model and weaker on browser automation. Playwright alone gives you the browser layer but nothing above it — no queue, no retries, no fingerprint stack. Colly in Go is fast and minimal but ignores the browser problem entirely. For a hosted, no-code option, Bright Data and Octoparse are credible — at a very different price point.

Verdict: Is Crawlee Worth It?

Yes — emphatically. If you're writing more than a one-off scraper in 2026, Crawlee is the default starting point. It is one of the very few open-source frameworks that hits the right level of abstraction: high enough to delete a week of yak-shaving, low enough that you can still drop down to raw Playwright when you need to. The fact that it's free, Apache 2.0, dual-language and backed by a profitable parent company (Apify) makes the long-term bet about as safe as open source gets. The 90/100 reflects exactly that — a near-best-in-class tool whose only real frictions are the learning curve and a handful of edge-case anti-bot scenarios that no library can fully solve on its own.

Frequently Asked Questions

Is Crawlee free?

Yes. Crawlee is open-source under Apache 2.0 and free to use anywhere, forever. You only pay if you choose to run your crawlers on Apify's managed cloud, which starts at $0/month with $5 in free credits.

What platforms does Crawlee support?

Crawlee runs on macOS, Windows and Linux as a Node.js (18+) or Python (3.9+) library, and ships official Docker images for headless deployment.

How does Crawlee compare to Scrapy?

Scrapy is a mature Python-only framework focused on HTTP crawling. Crawlee is dual-language (TypeScript and Python), has first-class browser support via Puppeteer and Playwright, and ships modern fingerprinting and autoscaling out of the box. For new browser-heavy projects, Crawlee is the better default; for huge HTTP-only legacy pipelines, Scrapy still holds up.

Is Crawlee open source?

Yes — Apache 2.0 licensed. Source on github.com/apify/crawlee (JavaScript / TypeScript) and github.com/apify/crawlee-python (Python).

Can Crawlee bypass Cloudflare?

The default fingerprinting and session-pool stack handles most off-the-shelf anti-bot setups. Strict Cloudflare Turnstile, DataDome or PerimeterX deployments still typically require a residential proxy and sometimes a stealth plugin on top of Crawlee — that's an industry reality, not a Crawlee-specific limitation.