llm-scraper

Turn any webpage into structured data using LLMs

What is llm-scraper?

Turn any webpage into structured data using LLMs

Overview of llm-scraper

What it is

LLM Scraper is a TypeScript library that extracts structured data from any webpage using large language models (LLMs). It is based on the Playwright framework and supports schemas defined with Zod or JSON Schema. The library provides full type-safety with TypeScript and offers multiple formatting modes for loading page content.

Key features

  • Supports GPT, Sonnet, Gemini, Llama, Qwen model series
  • Schemas defined with Zod or JSON Schema
  • Full type-safety with TypeScript
  • Based on Playwright framework
  • Streaming objects
  • Code-generation
  • Supports 6 formatting modes: html, raw_html, markdown, text, image, custom

Use cases

Extracting structured data from any webpage, such as scraping top stories from news sites, collecting product information, or gathering any structured content that can be defined with a schema. The library can be used in both single-run and streaming modes, and can generate reusable Playwright scripts via code-generation.

Getting started

  1. Install the required dependencies from npm:npm i zod playwright llm-scraper2. Initialize your LLM (example with OpenAI):npm i @ai-sdk/openai``````jsimport { openai } from '@ai-sdk/openai'const llm = openai('gpt-4o')3. Create a new scraper instance:jsimport LLMScraper from 'llm-scraper'const scraper = new LLMScraper(llm)

License and pricing

License: MIT. Pricing: open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/mishushakovllm-scraper.svg)](https://yunodb.com/tools/mishushakovllm-scraper?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/mishushakovllm-scraper?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/mishushakovllm-scraper.svg" alt="Featured on YunoDB" height="20"></a>