engine

2bit quants, kernels and runtime for consumer devices.

What is engine?

2bit quants, kernels and runtime for consumer devices.

Overview of engine

What it is

Cactus is a hybrid edge-cloud AI engine designed for mobile devices and wearables. It provides OpenAI-compatible APIs for text, speech, and vision. The engine is composed of a zero-copy computation graph (Cactus Graph), CPU/GPU kernels for various devices (Cactus Kernels), and a custom rotation-based quantization technique (Cactus Quants).

Key features

  • OpenAI-compatible APIs for text, speech, and vision
  • Zero-copy computation graph for efficient tensor operations
  • CPU/GPU kernels optimized for Apple, Samsung, Pixel, and other devices
  • Custom rotation-based quantization (Cactus Quants) from 4-bit to 1-bit
  • Hybrid edge-cloud inference with automatic cloud handoff based on local model confidence
  • Supports model conversion from HuggingFace via cactus convert
  • Bindings available for Swift, Kotlin, Flutter, React Native, Python, and Rust

Use cases

Cactus is intended for running AI models on consumer devices such as phones, tablets, wearables, and laptops. It supports chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, and vector indexing. The hybrid architecture allows hard queries to be routed to the cloud automatically.

Getting started

On macOS, install via Homebrew:brew install cactus-compute/cactus/cactuscactus runOn Linux (Ubuntu/Debian), clone the repository and run the setup script:git clone https://github.com/cactus-compute/cactus && cd cactussource ./setup

License and pricing

The license is not stated in the provided metadata or README. The project is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/cactus-computeengine.svg)](https://yunodb.com/tools/cactus-computeengine?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/cactus-computeengine?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/cactus-computeengine.svg" alt="Featured on YunoDB" height="20"></a>