datasets
π€ The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools
What is datasets?
π€ The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools
Overview of datasets
What it is
π€ Datasets is a lightweight library that provides one-line dataloaders for many public datasets and efficient data pre-processing. It allows downloading and pre-processing major public datasets (image, audio, text in 467 languages, 3D medical images, video, agent traces) from the HuggingFace Datasets Hub with a single command like load_dataset("rajpurkar/squad"). It also supports pre-processing local datasets in formats such as CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, and NIfTI.
Key features
- One-line dataset loading: Load AI-ready datasets from the Hugging Face Hub or local files with
load_dataset(). - Multiple formats: Native support for CSV, JSON, JSONL, Parquet, Arrow, XML, Text, Webdataset, and more.
- Multi-modal data: Built-in support for text, audio, image, video, PDF, and NIfTI (3D medical) data.
- Streaming mode: Stream datasets without downloading β iterate over data on-the-fly with
streaming=True(up to 100x faster with Xet backend). - Apache Arrow backend: Zero-copy memory-mapped storage that frees users from RAM limitations.
- Smart caching: Cached results are automatically reused, avoiding repeated processing.
- Multi-framework interoperability: Native conversion to/from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark.
Use cases
- Loading and preparing public datasets for machine learning model training or evaluation.
- Pre-processing data with operations like mapping, tokenization, and filtering.
- Streaming large datasets that exceed disk or memory capacity.
- Working with multi-modal data including audio, images, video, PDFs, and medical imaging.
- Loading datasets from local files (CSV, Parquet, etc.) or from Python objects (dict, list, pandas DataFrame, generator).
Getting started
pip install datasets
from datasets import load_dataset
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])
For more details, see the repository README.
License and pricing
Licensed under Apache-2.0. Open source and free to use.
Based on the project's public README and repository metadata.