datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

What is datachain?

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Overview of datachain

What it is

DataChain is a Python library for processing large file collections in S3, GCS, Azure and local storage. It allows running readers and models over images, video, audio and sensor recordings, then saving the results as typed, versioned datasets.

Key features

  • Query saved metadata to find relevant files, episodes and clips.
  • Build new training selections from existing labels, embeddings and features without repeating original processing.
  • Coding agents can inspect saved results and producing code, then continue work in a new session.
  • Handles parallel workers, download overlap and checkpoints.
  • File provides streaming access or a reusable local cache; delta updates process changed files and reuse saved results.
  • Supports LeRobot episodes, MCAP time windows and WebDataset samples with the same Python API.

Use cases

  • Training and evaluation: combine metadata and model outputs, select useful examples, and export datasets to training tools.
  • Physical AI and robotics: select recordings by sensor topics, extract video frames, and curate data for robot learning.
  • Data exploration: investigate model failures, find rare conditions and gaps in data coverage, and answer new questions from saved results.
  • Large inventories: discover millions of objects using storage metadata, before reading their contents.

Getting started

Install DataChain with video support: bash pip install 'datachain[video]' For a coding agent, install the skill and start a new agent session: bash datachain skill install --target claude --local # Or: --target codex, cursor, copilot, pi See the repository README for more details and Python pipeline examples.

License and pricing

DataChain is licensed under Apache-2.0 and is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/datachain-aidatachain.svg)](https://yunodb.com/tools/datachain-aidatachain?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/datachain-aidatachain?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/datachain-aidatachain.svg" alt="Featured on YunoDB" height="20"></a>