data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
What is data-juicer?
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
Overview of data-juicer
What it is
Data-Juicer is an open-source data processing system designed for foundation models. It provides a modular, composable infrastructure for cleaning, synthesizing, and analyzing data across the AI lifecycle. The project is hosted on GitHub and is licensed under Apache-2.0.
Key features
- Over 200 operators for text, image, audio, video, and multimodal data
- Recipe-first YAML pipelines that are reproducible, versionable, and shareable
- Composable operators that can be chained into complex workflows
- Hot-reload capability for iterating on operators without pipeline restarts
- Full-spectrum data intelligence for foundation models, agent systems, and RAG
- Production-ready performance with automatic operator fusion, adaptive parallelism, and CUDA acceleration
- Built-in observability with tracing for debugging and auditing
Use cases
Data-Juicer is used for deduplicating web-scale pre-training corpora, curating agent interaction traces, preparing domain-specific RAG indices, and processing data for pre-training, fine-tuning, reinforcement learning, and evaluation. It scales from a single laptop to thousand-node clusters.
Getting started
bashuv pip install py-data-juicerdj-process --config demos/process_simple/process.yaml
License and pricing
This project is open source under the Apache-2.0 license.
這是什麼
Data-Juicer 是一個專為基礎模型設計的開源資料處理系統。它提供模組化、可組合的基礎設施,用於在 AI 生命週期中清理、合成和分析資料。該專案託管於 GitHub,並採用 Apache-2.0 授權。
主要功能
- 超過 200 個運算子,支援文字、圖片、音訊、影片和多模態資料
- 以配方為優先的 YAML 管道,可重現、版本化並共享
- 可組合的運算子,能串聯成複雜的工作流程
- 熱重載功能,無需重啟管道即可迭代運算子
- 全光譜資料智慧,適用於基礎模型、代理系統和 RAG
- 生產級效能,具備自動運算子融合、自適應平行化和 CUDA 加速
- 內建可觀測性,提供除錯和稽核的追蹤功能
使用情境
Data-Juicer 用於去重網路規模的預訓練語料庫、整理代理互動軌跡、準備領域特定的 RAG 索引,以及處理預訓練、微調、強化學習和評估的資料。它能從單一筆電擴展到數千節點的叢集。
快速開始
bashuv pip install py-data-juicerdj-process --config demos/process_simple/process.yaml
授權與價格
本專案採用 Apache-2.0 開源授權。
Based on the project's public README and repository metadata.