distilabel

Overview of distilabel

What it is

Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers. Its goal is to accelerate AI development by quickly generating high-quality, diverse datasets based on verified research methodologies for generating and judging with AI feedback. The project is open-source and community-maintained; the original authors have moved on and a group of community collaborators now maintain the project, with latest fixes in the develop branch.

Key features

• Programmatic approach for building scalable pipelines for data generation and AI feedback.
• Integrates AI feedback from any LLM provider using one unified API.
• Supports a wide range of LLM providers via extras including Anthropic, Cohere, Groq, Hugging Face Inference Endpoints, Hugging Face Transformers, LiteLLM, llama.cpp, Mistral AI, Ollama, OpenAI, Vertex AI, vLLM, MLX, and more.
• Includes structured generation support via Outlines and Instructor extras.
• Provides data processing extras for scaling with Ray, sentence embeddings (faiss, sentence-transformers), text clustering (UMAP, Scikit-learn), and duplicate detection (minhash with datasketch, nltk).

Use cases

• Generating synthetic data and AI feedback for traditional predictive NLP tasks (classification, extraction) and generative/LLM scenarios (instruction following, dialogue generation, judging).
• Creating large-scale preference datasets, such as the 1M OpenHermesPreference dataset.
• Improving model performance by filtering data based on AI feedback, e.g., distilabeled Intel Orca DPO dataset and improved OpenHermes model.
• Building specific-task datasets using latest research papers, like the haiku DPO dataset.

Getting started

shpip install distilabel --upgradeRequires Python 3.9+. For LLM integrations, install with extras, e.g.:shpip install distilabel[hf-inference-endpoints] --upgradeThen run a pipeline example (see repository README for the full Python code).

License and pricing

Distilabel is open-source under the Apache-2.0 license.

Based on the project's public README and repository metadata.