pagesift
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
What is pagesift?
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
Overview of pagesift
What it is
pagesift is a Rust web scraper and crawler that converts URLs, PDFs, and Office files into clean Markdown or JSON. It can be used as a CLI, an HTTP API (compatible with Firecrawl v1/v2 and Jina Reader), an MCP server, or via Python and Node.js SDKs. It is self-hosted as a single binary.
Key features
- Input: URLs (with optional JavaScript rendering), HTML, PDF, Word (.docx), Excel (.xlsx), PowerPoint (.pptx), EPUB, CSV
- Output: Markdown, HTML, raw HTML, links, images, metadata (title, publication date), JSON; warnings for incomplete results
- Interfaces: CLI, Firecrawl v1/v2 HTTP API, Jina Reader
/r/endpoint, MCP (stdio or HTTP), Python, Node.js/TypeScript, Rust crate - Crawling: robots.txt and per-host rate limits by default, sitemaps, focused crawls by query, bounded memory (14–380 MB peak per worker in benchmarks)
- Platforms: Linux x86_64/aarch64 (glibc ≥ 2.28 and musl), macOS x86_64/arm64, Windows x86_64; Docker image
ghcr.io/yolow92/pagesift - Not for OCR of scanned PDFs or images; reads text layer only
Use cases
- Feeding web pages and documents to an LLM, RAG index, or AI agent with Markdown that preserves headings, lists, tables, and links
- Self-hosting a Firecrawl- or Jina Reader-style service on own machines with no per-page fees
- Mixed input: handling HTML, PDF, Office files, and e-books in a single call
- Running on small machines and containers with bounded memory sized from the cgroup limit
Getting started
bashcargo install pagesift --lockedpagesift scrape https://example.comOr via Docker:bashdocker run -p 3002:3002 ghcr.io/yolow92/pagesift:0.81.5curl localhost:3002/r/https://example.com
License and pricing
License: AGPL-3.0, or a commercial license.
Based on the project's public README and repository metadata.