webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
What is webclaw?
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Overview of webclaw
What it is
webclaw is a fast, local-first web content extraction tool built in Rust. It turns URLs into clean markdown, JSON, and LLM-ready context. It provides a CLI, a REST API, an MCP server, and SDKs for AI agents and RAG pipelines. The project is open source under the AGPL-3.0 license.
Key features
- Scrape a single page and return clean markdown or LLM-optimized text.
- Crawl multiple pages and extract structured data with CSS selectors or XPath.
- Extract data tables from pages as JSON arrays.
- Find elements by text or regex, and find similar elements.
- Include or exclude specific CSS selectors to keep only the main content.
- Self-hostable server and hosted API at webclaw.io.
Use cases
- Preparing web content for LLM context or RAG pipelines.
- Extracting structured data (e.g., product prices, table data) from websites.
- Connecting AI agents (Claude, Cursor, etc.) to web content via MCP.
- Automating web scraping tasks from the terminal or Python scripts.
Getting started
Install via Homebrew: bash brew tap 0xMassi/webclaw brew install webclaw Or via Docker: bash docker run --rm ghcr.io/0xmassi/webclaw https://example.com Or via Cargo: bash cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.7.4 --locked webclaw-cli For agent setup (Claude Code, Cursor, etc.): bash npx create-webclaw For Python: bash cd python && uv build --wheel && pip install dist/webclaw_cli-*.whl
License and pricing
The project is licensed under AGPL-3.0. Pricing is not stated for the hosted API; the CLI, MCP server, and extraction engine are open source.
Based on the project's public README and repository metadata.