docetl
A system for agentic LLM-powered data processing and ETL
What is docetl?
A system for agentic LLM-powered data processing and ETL
Overview of docetl
What it is
DocETL is a system for agentic LLM-powered data processing and ETL. It helps process large collections of data (structured and unstructured) with LLMs. Users write each operation in natural language, and DocETL provides operators (map, reduce, filter, and more) and orchestrates them, parallelizing work across data. It optimizes pipelines automatically by swapping models, rewriting prompts, decomposing operations, and replacing subtasks with code where possible to raise accuracy and cut cost. It returns tables that can be queried in a database. Without DocETL, users would write each LLM call manually and tune for accuracy, cost, and latency by hand. DocETL was created at the EPIC Data Lab and Data Systems and Foundations group at UC Berkeley.
Key features
- Natural language operation specification: users describe operations in plain language.
- Operators: map, reduce, filter, resolve, split, gather, extract, and more.
- Automatic pipeline optimization: swaps models, rewrites prompts, decomposes operations, and replaces subtasks with code to improve accuracy and reduce cost.
- Python API for production code, notebooks, and scripting.
- YAML low-code configuration for declarative pipelines.
- DocWrangler UI: a visual playground for interactive prompt development.
- Returns tables that can be queried in a database.
Use cases
- Processing large collections of data (structured and unstructured) with LLMs.
- Classifying support tickets and summarizing categories.
- Extracting specific information from documents, such as complaints from tickets.
- General ETL tasks involving LLM-powered data transformation.
Getting started
bash pip install docetl export OPENAI_API_KEY=your_key For more detailed instructions, see the repository README.
License and pricing
MIT License, open source.
Based on the project's public README and repository metadata.