sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

What is sail?

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Overview of sail

What it is

Sail is a drop-in Apache Spark replacement written in Rust. It unifies batch processing, stream processing, and compute-intensive AI workloads on a distributed, multimodal compute engine. It is compatible with the Spark Connect protocol, supporting the Spark SQL and DataFrame API with no code rewrites required. The engine is 100% Rust-native with no JVM overhead, delivering memory safety, instant startup, and predictable performance. According to the project's derived TPC-H benchmarks, Sail is approximately 10× faster than Spark and 98% cheaper on infrastructure costs. It has also been proven on ClickBench, outperforming Spark, popular Spark accelerators, Databricks, and Snowflake.

Key features

  • Drop-in Apache Spark replacement compatible with Spark Connect protocol (no code rewrites required).
  • 100% Rust-native engine with no JVM overhead, memory safety, instant startup, and predictable performance.
  • Columnar format and vectorized execution built on Apache Arrow and Apache DataFusion, enabling fast query execution via SIMD instructions.
  • Lightning-fast Python UDFs with zero serialization overhead; Arrow array pointers enable zero-copy data sharing.
  • Performant data shuffling where workers exchange Arrow columnar data directly, minimizing shuffle costs for joins and aggregations.
  • Lightweight, stateless workers that start in seconds, consume only a few megabytes of memory at idle, and scale elastically to cut cloud costs.
  • Native support for Delta Lake and Apache Iceberg table formats, and integration with catalog providers including Apache Iceberg REST Catalog, AWS Glue, Unity Catalog, Hive Metastore, and Microsoft OneLake.

Use cases

Sail is designed for batch processing, stream processing, and compute-intensive AI workloads. It can be used for ETL pipelines, machine learning, analytics, and any data processing task that traditionally runs on Apache Spark. Its compatibility with the Spark Connect protocol allows existing PySpark code to run without modification.

Getting started

Install Sail and PySpark client via pip: bash pip install pysail pip install pyspark-client Start the local Sail server using the command line: bash sail spark server --port 50051 Then connect from PySpark (no code changes needed): python from pyspark.sql import SparkSession spark = SparkSession.builder.remote('sc://localhost:50051').getOrCreate() spark.sql('SELECT 1 + 1').show() For Kubernetes deployment, see the repository README.

License and pricing

Sail is open source under the Apache-2.0 license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/lakehqsail.svg)](https://yunodb.com/tools/lakehqsail?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/lakehqsail?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/lakehqsail.svg" alt="Featured on YunoDB" height="20"></a>