lance
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible wi
What is lance?
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
Overview of lance
What it is
Lance is an open lakehouse format for multimodal AI. It provides a file format, table format, and catalog specification that enables building a complete lakehouse on object storage for AI workflows. It is compatible with Pandas, DuckDB, Polars, PyArrow, Ray, Spark, and more integrations.
Key features
- Expressive hybrid search: Combine vector similarity search, full-text search (BM25), and SQL analytics on the same dataset with accelerated secondary indices.
- Lightning-fast random access: 100x faster than Parquet or Iceberg for random access without sacrificing scan performance.
- Native multimodal data support: Store images, videos, audio, text, and embeddings in a single unified format with efficient blob encoding and lazy loading.
- Data evolution: Efficiently add columns with backfilled values without full table rewrites, suitable for ML feature engineering.
- Zero-copy versioning: Automatic versioning with ACID transactions, time travel, tags, and branches—no extra infrastructure needed.
- Rich ecosystem integrations: Apache Arrow, Pandas, Polars, DuckDB, Apache Spark, Ray, Trino, Apache Flink, and open catalogs (Apache Polaris, Unity Catalog, Apache Gravitino).
Use cases
- Building search engines and feature stores with hybrid search capabilities.
- Large-scale ML training requiring high performance IO and random access.
- Storing, querying, and managing multimodal data including images, videos, audio, text, and embeddings.
Getting started
Install the Python package: shell pip install pylance To install a preview release: shell pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance Convert a Parquet dataset to Lance: python import lance import pandas as pd import pyarrow as pa import pyarrow.dataset df = pd.DataFrame({'a': [5], 'b': [10]}) uri = '/tmp/test.parquet' tbl = pa.Table.from_pandas(df) pa.dataset.write_dataset(tbl, uri, format='parquet') parquet = pa.dataset.dataset(uri, format='parquet') lance.write_dataset(parquet, '/tmp/test.lance') Read Lance data: python dataset = lance.dataset('/tmp/test.lance') assert isinstance(dataset, pa.dataset.Dataset)
License and pricing
Apache-2.0 license, open source.
Based on the project's public README and repository metadata.