tiny-llm
learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen
What is tiny-llm?
learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen
Overview of tiny-llm
What it is
tiny-llm is a hands-on course for systems engineers who want to understand LLM inference end to end. It builds the path that loads a Qwen3 model, turns tokens into logits, and generates text. The course is built on MLX arrays and the MLX extension runtime, without using high-level neural-network layers. When a chapter teaches an operator, the student implements that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX remains the correctness oracle and performance baseline.
Key features
- Follows a four-week learning path: Week 1 builds a Qwen3 model from array operations; Week 2 adds caching, quantization, and fused primitives; Week 3 builds a mini vLLM with continuous batching and paged KV cache; Week 4 builds a coding agent.
- Uses Apple Silicon as the target platform, providing a practical local environment with one shared memory space and direct access to Metal kernels.
- Uses Qwen3-4B, which is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally.
- The course is published at skyzh.github.io/tiny-llm.
- The
tiny_llmpackage is where students implement the exercises;tiny_llm_refcontains the reference solution. - Community port: amagilabs/tiny-llm-cuda is an unofficial CUDA (NVIDIA GPU) backend for the reference solution kernels.
Use cases
- Systems engineers who want to understand LLM inference end to end.
- Learning how to build an LLM-serving system on Apple Silicon.
- Understanding concepts such as attention, KV-cache, continuous batching, paged attention, and agent loops.
Getting started
See the repository README. The book is published at skyzh.github.io/tiny-llm. To verify an existing checkout, run:bashpdm install -vpdm run check-installationpdm run test-refsol -- -- -k week_1
License and pricing
License: Apache-2.0. Pricing: open source.
Based on the project's public README and repository metadata.