llm-scraper
Turn any webpage into structured data using LLMs
What is llm-scraper?
Turn any webpage into structured data using LLMs
Overview of llm-scraper
What it is
LLM Scraper is a TypeScript library that extracts structured data from any webpage using large language models (LLMs). It is based on the Playwright framework and supports schemas defined with Zod or JSON Schema. The library provides full type-safety with TypeScript and offers multiple formatting modes for loading page content.
Key features
- Supports GPT, Sonnet, Gemini, Llama, Qwen model series
- Schemas defined with Zod or JSON Schema
- Full type-safety with TypeScript
- Based on Playwright framework
- Streaming objects
- Code-generation
- Supports 6 formatting modes: html, raw_html, markdown, text, image, custom
Use cases
Extracting structured data from any webpage, such as scraping top stories from news sites, collecting product information, or gathering any structured content that can be defined with a schema. The library can be used in both single-run and streaming modes, and can generate reusable Playwright scripts via code-generation.
Getting started
- Install the required dependencies from npm:
npm i zod playwright llm-scraper2. Initialize your LLM (example with OpenAI):npm i @ai-sdk/openai``````jsimport { openai } from '@ai-sdk/openai'const llm = openai('gpt-4o')3. Create a new scraper instance:jsimport LLMScraper from 'llm-scraper'const scraper = new LLMScraper(llm)
License and pricing
License: MIT. Pricing: open source.
Based on the project's public README and repository metadata.