SwiftLM

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS +

What is SwiftLM?

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

Overview of SwiftLM

What it is

SwiftLM is a native Swift inference server that serves MLX models with an OpenAI-compatible API. It runs on Apple Silicon, compiled to a single binary with no Python runtime, no Global Interpreter Lock (GIL), and no unnecessary memory copies. The server supports SSD streaming for large Mixture-of-Experts (MoE) models and TurboQuant KV cache compression.

Key features

  • OpenAI-compatible API endpoint - SSD streaming for 100B+ MoE models (via --stream-experts flag)
  • TurboQuant KV cache compression - Native Swift, no Python runtime, single binary - Metal kernel compilation for Apple Silicon - Prompt cache support - Pre-built binary available for macOS arm64

Use cases

  • Running large language models on Apple Silicon hardware, especially MoE models that benefit from SSD streaming - Providing a local or edge inference server with an OpenAI-compatible API - Testing and benchmarking MLX models on macOS (e.g., Mac mini M6)

Getting started

Download the pre-built binary and run: bash tar -xzf SwiftLM--macos-arm64.tar.gz ./SwiftLM --model mlx-community/Qwen2.5-3B-Instruct-4bit --port 5413 Or build from source: bash git clone --recursive https://github.com/SharpAI/SwiftLM cd SwiftLM ./build.sh .build/release/SwiftLM --model mlx-community/gemma-4-26b-a4b-it-4bit --port 5413 (Add --stream-experts for oversized MoE models to stream expert layers from NVMe SSD.)

License and pricing

The project is open source under the MIT license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/sharpaiswiftlm.svg)](https://yunodb.com/tools/sharpaiswiftlm?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/sharpaiswiftlm?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/sharpaiswiftlm.svg" alt="Featured on YunoDB" height="20"></a>