tiny-llm

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

What is tiny-llm?

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

Overview of tiny-llm

What it is

tiny-llm is a hands-on course for systems engineers who want to understand LLM inference end to end. It builds the path that loads a Qwen3 model, turns tokens into logits, and generates text. The course is built on MLX arrays and the MLX extension runtime, without using high-level neural-network layers. When a chapter teaches an operator, the student implements that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX remains the correctness oracle and performance baseline.

Key features

  • Follows a four-week learning path: Week 1 builds a Qwen3 model from array operations; Week 2 adds caching, quantization, and fused primitives; Week 3 builds a mini vLLM with continuous batching and paged KV cache; Week 4 builds a coding agent.
  • Uses Apple Silicon as the target platform, providing a practical local environment with one shared memory space and direct access to Metal kernels.
  • Uses Qwen3-4B, which is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally.
  • The course is published at skyzh.github.io/tiny-llm.
  • The tiny_llm package is where students implement the exercises; tiny_llm_ref contains the reference solution.
  • Community port: amagilabs/tiny-llm-cuda is an unofficial CUDA (NVIDIA GPU) backend for the reference solution kernels.

Use cases

  • Systems engineers who want to understand LLM inference end to end.
  • Learning how to build an LLM-serving system on Apple Silicon.
  • Understanding concepts such as attention, KV-cache, continuous batching, paged attention, and agent loops.

Getting started

See the repository README. The book is published at skyzh.github.io/tiny-llm. To verify an existing checkout, run:bashpdm install -vpdm run check-installationpdm run test-refsol -- -- -k week_1

License and pricing

License: Apache-2.0. Pricing: open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/skyzhtiny-llm.svg)](https://yunodb.com/tools/skyzhtiny-llm?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/skyzhtiny-llm?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/skyzhtiny-llm.svg" alt="Featured on YunoDB" height="20"></a>