webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

What is webclaw?

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Overview of webclaw

What it is

webclaw is a fast, local-first web content extraction tool built in Rust. It turns URLs into clean markdown, JSON, and LLM-ready context. It provides a CLI, a REST API, an MCP server, and SDKs for AI agents and RAG pipelines. The project is open source under the AGPL-3.0 license.

Key features

  • Scrape a single page and return clean markdown or LLM-optimized text.
  • Crawl multiple pages and extract structured data with CSS selectors or XPath.
  • Extract data tables from pages as JSON arrays.
  • Find elements by text or regex, and find similar elements.
  • Include or exclude specific CSS selectors to keep only the main content.
  • Self-hostable server and hosted API at webclaw.io.

Use cases

  • Preparing web content for LLM context or RAG pipelines.
  • Extracting structured data (e.g., product prices, table data) from websites.
  • Connecting AI agents (Claude, Cursor, etc.) to web content via MCP.
  • Automating web scraping tasks from the terminal or Python scripts.

Getting started

Install via Homebrew: bash brew tap 0xMassi/webclaw brew install webclaw Or via Docker: bash docker run --rm ghcr.io/0xmassi/webclaw https://example.com Or via Cargo: bash cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.7.4 --locked webclaw-cli For agent setup (Claude Code, Cursor, etc.): bash npx create-webclaw For Python: bash cd python && uv build --wheel && pip install dist/webclaw_cli-*.whl

License and pricing

The project is licensed under AGPL-3.0. Pricing is not stated for the hosted API; the CLI, MCP server, and extraction engine are open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/0xmassiwebclaw.svg)](https://yunodb.com/tools/0xmassiwebclaw?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/0xmassiwebclaw?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/0xmassiwebclaw.svg" alt="Featured on YunoDB" height="20"></a>