crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this

What is crawl4ai?

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

Overview of crawl4ai

What it is

Crawl4AI is an open-source web crawler and scraper designed for LLMs and AI agents. It converts any website into clean, LLM-ready Markdown. Users can run it themselves (free forever) or use the hosted Crawl4AI Cloud with a single API key. The project is licensed under Apache-2.0 and is categorized under AI.

Key features

  • Markdown generation: produces clean Markdown with headings, lists, tables, code blocks, and citation hints; includes filters like PruningContentFilterLXML, BM25ContentFilter, and LLMContentFilter.
  • Structured data extraction: supports CSS/XPath schemas, LLM extraction (any provider), schema generator, chunking (topic, regex, sentence), and cosine similarity search.
  • Browser control: persistent profiles, remote browsers via CDP, sessions, proxies with auth and rotation, stealth mode, support for Chromium, Firefox, and WebKit.
  • Deep crawling: BFS, DFS, best-first strategies, crash recovery (resume_state), adaptive crawling (AdaptiveCrawler), URL discovery (AsyncUrlSeeder, DomainMapper), dynamic page handling (JS, infinite scroll).
  • Self-hosting (Docker): REST API endpoints (/md, /html, /crawl, /screenshot, etc.), MCP support, monitoring dashboard, AMD64 and ARM64 images.
  • Cloud-specific features: web search API (GET /search), SERP API (GET /serp), answer API (GET /answer), extraction without own LLM key, automatic handling of JS-heavy pages and bot walls, MCP for agents.

Use cases

  • RAG (Retrieval-Augmented Generation) pipelines.
  • AI agents that need to fetch and process web content.
  • Data extraction for analytics, research, or monitoring.
  • Building custom search or answer systems.
  • Large-scale crawling with crash recovery and adaptive strategies.

Getting started

bash pip install -U crawl4ai crawl4ai-setup # installs the browser, once python import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url='https://news.ycombinator.com') print(result.markdown) asyncio.run(main()) For Docker server: bash export CRAWL4AI_API_TOKEN='$(openssl rand -hex 32)' docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g -e CRAWL4AI_API_TOKEN='$CRAWL4AI_API_TOKEN' unclecode/craw See the repository README for full installation options.

License and pricing

The project is open-source under the Apache-2.0 license. The self-hosted version is free forever. Crawl4AI Cloud offers a free tier (10 requests per minute, 2 concurrent, 2,000 credits to start) and paid plans (120 requests per minute, 10 concurrent, from $10).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/unclecodecrawl4ai.svg)](https://yunodb.com/tools/unclecodecrawl4ai?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/unclecodecrawl4ai?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/unclecodecrawl4ai.svg" alt="Featured on YunoDB" height="20"></a>