slothdb
An experimental embedded SQL engine in C++20. Query Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly with SQL, in-process. Early-stage.
What is slothdb?
An experimental embedded SQL engine in C++20. Query Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly with SQL, in-process. Early-stage.
Overview of slothdb
What it is
SlothDB is an experimental embedded SQL database engine written in C++20. It is in early development. It allows querying Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly with SQL, in-process, similar to DuckDB and SQLite. The project is a learning exercise and is early-stage; performance numbers should be treated as anecdotal until architecture work lands.
Key features
- Query multiple file formats (Parquet, CSV, JSON, Arrow, Avro, SQLite, Excel) directly with SQL, in-process.
- Natural language query support via a rules parser (sub-10 ms, no model) and an optional local Qwen2.5-Coder model (0.5B or 1.5B) for generating SQL from natural language.
- Python, Node/JS, and shell interfaces; Python wheels available for Linux, macOS, Windows.
- HTTP range requests for Parquet files, allowing partial downloads.
- Predicate pushdown using row-group statistics to skip irrelevant row groups.
- Top-N pushdown for ORDER BY ... LIMIT queries using a bounded heap.
- Nested aggregate functions (e.g., ROUND(AVG(x))) and ORDER BY by aggregate alias.
- Arithmetic type coercion and promotion.
- GZIP and Zstd Parquet decoding support.
- Glob pattern support for reading multiple files (e.g.,
data/*.parquet). - Single-threaded WebAssembly build for browser playground.
- HTTPS Parquet fetching via a Cloudflare CORS proxy.
- Discord community for bug reports and discussion.
Use cases
- Ad-hoc querying of data files (Parquet, CSV, JSON, etc.) without loading into a database.
- Embedding a lightweight SQL engine into applications for data analysis.
- Natural language querying for non-technical users via the
.asksub-command. - Quick data exploration and prototyping in Python, Node.js, or the shell.
Getting started
Install the Python package and run the demo: bash pip install slothdb python -c 'import slothdb; slothdb.demo()' For Node.js: bash npm install @slothdb/wasm For the shell, download a prebuilt binary or build from source; see the repository for instructions.
License and pricing
Licensed under the MIT License. Open source and free.
Based on the project's public README and repository metadata.