crawl4ai
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this
What is crawl4ai?
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Overview of crawl4ai
What it is
Crawl4AI is an open-source web crawler and scraper designed for LLMs and AI agents. It converts any website into clean, LLM-ready Markdown. Users can run it themselves (free forever) or use the hosted Crawl4AI Cloud with a single API key. The project is licensed under Apache-2.0 and is categorized under AI.
Key features
- Markdown generation: produces clean Markdown with headings, lists, tables, code blocks, and citation hints; includes filters like PruningContentFilterLXML, BM25ContentFilter, and LLMContentFilter.
- Structured data extraction: supports CSS/XPath schemas, LLM extraction (any provider), schema generator, chunking (topic, regex, sentence), and cosine similarity search.
- Browser control: persistent profiles, remote browsers via CDP, sessions, proxies with auth and rotation, stealth mode, support for Chromium, Firefox, and WebKit.
- Deep crawling: BFS, DFS, best-first strategies, crash recovery (resume_state), adaptive crawling (AdaptiveCrawler), URL discovery (AsyncUrlSeeder, DomainMapper), dynamic page handling (JS, infinite scroll).
- Self-hosting (Docker): REST API endpoints (/md, /html, /crawl, /screenshot, etc.), MCP support, monitoring dashboard, AMD64 and ARM64 images.
- Cloud-specific features: web search API (GET /search), SERP API (GET /serp), answer API (GET /answer), extraction without own LLM key, automatic handling of JS-heavy pages and bot walls, MCP for agents.
Use cases
- RAG (Retrieval-Augmented Generation) pipelines.
- AI agents that need to fetch and process web content.
- Data extraction for analytics, research, or monitoring.
- Building custom search or answer systems.
- Large-scale crawling with crash recovery and adaptive strategies.
Getting started
bash pip install -U crawl4ai crawl4ai-setup # installs the browser, once python import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url='https://news.ycombinator.com') print(result.markdown) asyncio.run(main()) For Docker server: bash export CRAWL4AI_API_TOKEN='$(openssl rand -hex 32)' docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g -e CRAWL4AI_API_TOKEN='$CRAWL4AI_API_TOKEN' unclecode/craw See the repository README for full installation options.
License and pricing
The project is open-source under the Apache-2.0 license. The self-hosted version is free forever. Crawl4AI Cloud offers a free tier (10 requests per minute, 2 concurrent, 2,000 credits to start) and paid plans (120 requests per minute, 10 concurrent, from $10).
Based on the project's public README and repository metadata.