Daft
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
What is Daft?
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Overview of Daft
What it is
Daft is a high-performance data engine for AI and multimodal workloads. It processes images, audio, video, and structured data at any scale. The engine is Python-native with Rust under the hood, and can scale from local to distributed clusters on Ray or Kubernetes. It is licensed under Apache 2.0 and is open source.
Key features
- Native multimodal processing: Process images, audio, video, and embeddings alongside structured data in a single framework - Built-in AI operations: Run LLM prompts, generate embeddings, and classify data at scale using OpenAI, Transformers, or custom models - Python-native, Rust-powered: Skip the JVM complexity with Python at its core and Rust under the hood for blazing performance - Seamless scaling: Start local, scale to distributed clusters on Ray, Kubernetes - Universal connectivity: Access data anywhere (S3, GCS, Iceberg, Delta Lake, Hugging Face, Unity Catalog)
- Out-of-box reliability: Intelligent memory management and sensible defaults eliminate configuration headaches
Use cases
Daft is designed for AI and multimodal data pipelines. Use cases include processing large volumes of images, audio, video, and structured data for machine learning; running LLM prompts and generating embeddings at scale; building data preprocessing workflows for AI training; and performing classification or inference on multimodal datasets.
Getting started
Install Daft with pip: pip install daft Requires Python 3.10 or higher. For a quickstart with a real-world e-commerce dataset, see the repository README.
License and pricing
Licensed under Apache-2.0. Open source.
Based on the project's public README and repository metadata.