arkflow

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

What is arkflow?

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

Overview of arkflow

What it is

ArkFlow is a high performance Rust stream processing engine that integrates AI capabilities. It provides real-time data processing and intelligent analysis. It supports multiple input/output sources and processors, and enables loading and execution of machine learning models for streaming data inference, anomaly detection, and complex event processing. ArkFlow is enlisted in the CNCF Cloud Native Landscape.

Key features

  • High Performance: Built on Rust and Tokio async runtime, offering performance and low latency
  • Durable Delivery: At-least-once by default via per-stream WAL durability, with optional exactly-once for transactional sinks
  • Multiple Data Sources: Support for Kafka, MQTT, HTTP, files, SQL databases, and many other input/output sources
  • Powerful Processing: Built-in SQL queries, Python UDFs, JSON processing, Protobuf encoding/decoding, batch processing, and VRL
  • Streaming Codecs: JSON and Protobuf codecs, plus Debezium CDC envelopes and Confluent Schema Registry wire-format
  • Control Plane: An optional Hub and web console to observe, configure, and operate multiple ArkFlow compute nodes as a fleet
  • Extensible: Modular design, easy to extend with new input, buffer, output, and processor components

Use cases

ArkFlow can be used for real-time data processing pipelines that require high throughput and low latency. It is suitable for scenarios such as streaming data inference with machine learning models, anomaly detection, complex event processing, and integrating AI capabilities into data streams. It can ingest data from various sources like Kafka, MQTT, HTTP, files, and SQL databases, process it with SQL, Python UDFs, or AI models, and output to targets like Kafka, InfluxDB, MongoDB, or standard output.

Getting started

bash# Clone the repositorygit clone https://github.com/arkflow-rs/arkflow.gitcd arkflow# Build the projectcargo build --release# Run testscargo testCreate a configuration file config.yaml as described in the repository README, then run:bash./target/release/arkflow --config config.yaml

License and pricing

License: Apache-2.0. Pricing: open source (free).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/arkflow-rsarkflow.svg)](https://yunodb.com/tools/arkflow-rsarkflow?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/arkflow-rsarkflow?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/arkflow-rsarkflow.svg" alt="Featured on YunoDB" height="20"></a>