pagesift

Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.

What is pagesift?

Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.

Overview of pagesift

What it is

pagesift is a Rust web scraper and crawler that converts URLs, PDFs, and Office files into clean Markdown or JSON. It can be used as a CLI, an HTTP API (compatible with Firecrawl v1/v2 and Jina Reader), an MCP server, or via Python and Node.js SDKs. It is self-hosted as a single binary.

Key features

  • Input: URLs (with optional JavaScript rendering), HTML, PDF, Word (.docx), Excel (.xlsx), PowerPoint (.pptx), EPUB, CSV
  • Output: Markdown, HTML, raw HTML, links, images, metadata (title, publication date), JSON; warnings for incomplete results
  • Interfaces: CLI, Firecrawl v1/v2 HTTP API, Jina Reader /r/ endpoint, MCP (stdio or HTTP), Python, Node.js/TypeScript, Rust crate
  • Crawling: robots.txt and per-host rate limits by default, sitemaps, focused crawls by query, bounded memory (14–380 MB peak per worker in benchmarks)
  • Platforms: Linux x86_64/aarch64 (glibc ≥ 2.28 and musl), macOS x86_64/arm64, Windows x86_64; Docker image ghcr.io/yolow92/pagesift
  • Not for OCR of scanned PDFs or images; reads text layer only

Use cases

  • Feeding web pages and documents to an LLM, RAG index, or AI agent with Markdown that preserves headings, lists, tables, and links
  • Self-hosting a Firecrawl- or Jina Reader-style service on own machines with no per-page fees
  • Mixed input: handling HTML, PDF, Office files, and e-books in a single call
  • Running on small machines and containers with bounded memory sized from the cgroup limit

Getting started

bashcargo install pagesift --lockedpagesift scrape https://example.comOr via Docker:bashdocker run -p 3002:3002 ghcr.io/yolow92/pagesift:0.81.5curl localhost:3002/r/https://example.com

License and pricing

License: AGPL-3.0, or a commercial license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/yolow92pagesift.svg)](https://yunodb.com/tools/yolow92pagesift?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/yolow92pagesift?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/yolow92pagesift.svg" alt="Featured on YunoDB" height="20"></a>