rkllama
Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )
What is rkllama?
Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )
Overview of rkllama
What it is
rkllama is a server and client for running and interacting with large language models (LLMs) optimized for Rockchip RK3588(S) and RK3576 platforms. It is an alternative to Ollama and Llama.cpp that allows models to run on the Rockchip NPU (Neural Processing Unit). The project uses the rkllm-runtime library (version 1.3.0) and rknn-runtime library (version 2.3.2). It supports Python versions 3.9 to 3.12 and has been tested on hardware such as Orange Pi 5 Pro, Orange Pi 5 Plus, Orange Pi 5 Max, and Radxa Rock 4d, running Ubuntu 24.04 or Armbian Linux.
Key features
- NPU-accelerated inference for LLMs on Rockchip devices.
- Ollama API compatibility: endpoints
/api/chat,/api/generate,/api/ps,/api/tags,/api/embed,/api/version,/api/pull. - Partial OpenAI API compatibility: endpoints
/v1/completions,/v1/chat/completions,/v1/embeddings,/v1/images/generations,/v1/audio/speech,/v1/audio/transcriptions,/v1/audio/translations. - Tool/Function Calling support with multiple LLM formats (Qwen, Llama 3.2+, others).
- Dynamic loading and unloading of models with automatic prompt cache saving and restoration.
- Multimodal support: vision models (Qwen2VL, Qwen2.5VL, Qwen3VL, MiniCPMV4, MiniCPMV4.5, InternVL3.5), image generation (LCM Stable Diffusion 1.5 RKNN), text-to-speech (Piper TTS, MMS-TTS), and speech-to-text (omniASR-CTC, Whisper).
- Support for .GGUF models through a llama.cpp fork (rk-llama.cpp).
Use cases
- Running LLMs on Rockchip-based single-board computers (Orange Pi, Radxa) with NPU acceleration.
- Deploying a local AI server for chat, text generation, embeddings, and audio tasks.
- Integrating with tools like OpenWebUI or custom applications via Ollama-compatible or OpenAI-compatible APIs.
- Using vision models to answer questions about images, generating images from text, and performing speech-to-text or text-to-speech.
- Experimenting with tool/function calling for agent-based workflows.
Getting started
Standard installation: bash git clone https://github.com/notpunchnox/rkllama cd rkllama python -m pip install . Start the server: bash rkllama_server --models --llamacpp Start the client: bash rkllama_client For Docker installation: bash docker pull ghcr.io/notpunchnox/rkllama:main docker run -it --privileged -p 8080:8080 -v :/opt/rkllama/models ghcr.io/notpunchnox/rkllama:main
License and pricing
The project is open source under the GPL-3.0 license.
Based on the project's public README and repository metadata.
