data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

What is data-juicer?

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

Overview of data-juicer

What it is

Data-Juicer is an open-source data processing system designed for foundation models. It provides a modular, composable infrastructure for cleaning, synthesizing, and analyzing data across the AI lifecycle. The project is hosted on GitHub and is licensed under Apache-2.0.

Key features

  • Over 200 operators for text, image, audio, video, and multimodal data
  • Recipe-first YAML pipelines that are reproducible, versionable, and shareable
  • Composable operators that can be chained into complex workflows
  • Hot-reload capability for iterating on operators without pipeline restarts
  • Full-spectrum data intelligence for foundation models, agent systems, and RAG
  • Production-ready performance with automatic operator fusion, adaptive parallelism, and CUDA acceleration
  • Built-in observability with tracing for debugging and auditing

Use cases

Data-Juicer is used for deduplicating web-scale pre-training corpora, curating agent interaction traces, preparing domain-specific RAG indices, and processing data for pre-training, fine-tuning, reinforcement learning, and evaluation. It scales from a single laptop to thousand-node clusters.

Getting started

bashuv pip install py-data-juicerdj-process --config demos/process_simple/process.yaml

License and pricing

This project is open source under the Apache-2.0 license.

這是什麼

Data-Juicer 是一個專為基礎模型設計的開源資料處理系統。它提供模組化、可組合的基礎設施,用於在 AI 生命週期中清理、合成和分析資料。該專案託管於 GitHub,並採用 Apache-2.0 授權。

主要功能

  • 超過 200 個運算子,支援文字、圖片、音訊、影片和多模態資料
  • 以配方為優先的 YAML 管道,可重現、版本化並共享
  • 可組合的運算子,能串聯成複雜的工作流程
  • 熱重載功能,無需重啟管道即可迭代運算子
  • 全光譜資料智慧,適用於基礎模型、代理系統和 RAG
  • 生產級效能,具備自動運算子融合、自適應平行化和 CUDA 加速
  • 內建可觀測性,提供除錯和稽核的追蹤功能

使用情境

Data-Juicer 用於去重網路規模的預訓練語料庫、整理代理互動軌跡、準備領域特定的 RAG 索引,以及處理預訓練、微調、強化學習和評估的資料。它能從單一筆電擴展到數千節點的叢集。

快速開始

bashuv pip install py-data-juicerdj-process --config demos/process_simple/process.yaml

授權與價格

本專案採用 Apache-2.0 開源授權。

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/datajuicerdata-juicer.svg)](https://yunodb.com/tools/datajuicerdata-juicer?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/datajuicerdata-juicer?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/datajuicerdata-juicer.svg" alt="Featured on YunoDB" height="20"></a>