datasets

πŸ€— The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

What is datasets?

πŸ€— The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

Overview of datasets

What it is

πŸ€— Datasets is a lightweight library that provides one-line dataloaders for many public datasets and efficient data pre-processing. It allows downloading and pre-processing major public datasets (image, audio, text in 467 languages, 3D medical images, video, agent traces) from the HuggingFace Datasets Hub with a single command like load_dataset("rajpurkar/squad"). It also supports pre-processing local datasets in formats such as CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, and NIfTI.

Key features

  • One-line dataset loading: Load AI-ready datasets from the Hugging Face Hub or local files with load_dataset().
  • Multiple formats: Native support for CSV, JSON, JSONL, Parquet, Arrow, XML, Text, Webdataset, and more.
  • Multi-modal data: Built-in support for text, audio, image, video, PDF, and NIfTI (3D medical) data.
  • Streaming mode: Stream datasets without downloading β€” iterate over data on-the-fly with streaming=True (up to 100x faster with Xet backend).
  • Apache Arrow backend: Zero-copy memory-mapped storage that frees users from RAM limitations.
  • Smart caching: Cached results are automatically reused, avoiding repeated processing.
  • Multi-framework interoperability: Native conversion to/from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark.

Use cases

  • Loading and preparing public datasets for machine learning model training or evaluation.
  • Pre-processing data with operations like mapping, tokenization, and filtering.
  • Streaming large datasets that exceed disk or memory capacity.
  • Working with multi-modal data including audio, images, video, PDFs, and medical imaging.
  • Loading datasets from local files (CSV, Parquet, etc.) or from Python objects (dict, list, pandas DataFrame, generator).

Getting started

pip install datasets
from datasets import load_dataset

squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])

For more details, see the repository README.

License and pricing

Licensed under Apache-2.0. Open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/huggingfacedatasets.svg)](https://yunodb.com/tools/huggingfacedatasets?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/huggingfacedatasets?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/huggingfacedatasets.svg" alt="Featured on YunoDB" height="20"></a>