datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
What is datachain?
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Overview of datachain
What it is
DataChain is a Python library for processing large file collections in S3, GCS, Azure and local storage. It allows running readers and models over images, video, audio and sensor recordings, then saving the results as typed, versioned datasets.
Key features
- Query saved metadata to find relevant files, episodes and clips.
- Build new training selections from existing labels, embeddings and features without repeating original processing.
- Coding agents can inspect saved results and producing code, then continue work in a new session.
- Handles parallel workers, download overlap and checkpoints.
- File provides streaming access or a reusable local cache; delta updates process changed files and reuse saved results.
- Supports LeRobot episodes, MCAP time windows and WebDataset samples with the same Python API.
Use cases
- Training and evaluation: combine metadata and model outputs, select useful examples, and export datasets to training tools.
- Physical AI and robotics: select recordings by sensor topics, extract video frames, and curate data for robot learning.
- Data exploration: investigate model failures, find rare conditions and gaps in data coverage, and answer new questions from saved results.
- Large inventories: discover millions of objects using storage metadata, before reading their contents.
Getting started
Install DataChain with video support: bash pip install 'datachain[video]' For a coding agent, install the skill and start a new agent session: bash datachain skill install --target claude --local # Or: --target codex, cursor, copilot, pi See the repository README for more details and Python pipeline examples.
License and pricing
DataChain is licensed under Apache-2.0 and is open source.
Based on the project's public README and repository metadata.