dungbeetle

Overview of dungbeetle

What it is

DungBeetle is a lightweight, single binary distributed job server designed for queuing and asynchronously executing large numbers of SQL read jobs (e.g., reports) against SQL databases. When the read jobs are executed, the results are written to separate ephemeral results databases, where the results of every job is its own dedicated table, enabling faster retrieval. It is built by Zerodha.

Key features

• Supports MySQL, PostgreSQL, and ClickHouse as source databases.
• Supports MySQL and PostgreSQL as result/cache databases for job results.
• Standalone server that exposes HTTP APIs for managing jobs and groups of jobs (list, post, status check, cancel jobs).
• Reads SQL queries from .sql files and registers them as jobs ready to be queued.
• Supports multi-process, multi-threaded, asynchronous distributed job queueing via a common broker backend (Redis, AMQP, etc.).

Use cases

A prominent use case is user-facing report generation in applications with large SQL databases. When thousands of concurrent users request reports, every second of I/O delay during query execution can bottleneck the application and the database. DungBeetle defers report generation requests to a job queue, immediately freeing up the frontend application. Reports are presented to users asynchronously, and the server/worker instances act as traffic control to prevent the primary database from being inundated. Once reports are generated, subsequent queries for slicing, sorting, or filtering can be sent to the smaller, faster results cache database.

Getting started

See the repository README for installation instructions. Pre-compiled binaries are available from the releases page. Configuration is done via a config.toml file. SQL query tasks are written in .sql files in goyesql format. The server is started with:shelldungbeetle --config /path/to/config.toml --sql-directory /path/to/your/sql/queries# Run 'dungbeetle --help' to see all supported arguments

License and pricing

The license is MIT. It is open source.

Based on the project's public README and repository metadata.