diy-llm

Covers pre-training data, Tokenizer, Transformer, MoE,distributed training, Scaling Laws, inference & alignment .6 progressive code assignments for full-stack L

What is diy-llm?

Covers pre-training data, Tokenizer, Transformer, MoE,distributed training, Scaling Laws, inference & alignment .6 progressive code assignments for full-stack LLM learning | 涵盖预训练数据、分词器、Transformer、MoE、分布式训练、缩放定律、推理与对齐,6 项渐进代码作业,掌握 LLM 全栈知识

Overview of diy-llm

What it is

Diy-LLM is a community-driven Chinese tutorial project that re-organizes the knowledge system of building large language models (LLMs) from scratch. It is based on Stanford CS336: Language Modeling from Scratch (Spring 2025 and 2026) but is not a direct translation. The project covers the entire pipeline: tokenizer, data, training, system optimization, inference, alignment, evaluation, and multi-modality. It includes 16 chapters of text and 6 progressive coding assignments, all written and reviewed by human contributors with AI assistance for translation and organization only. The project is open source and hosted on GitHub under the Datawhale community.

Key features

  • 16 chapters and 6 assignments covering the full LLM pipeline from tokenizer to multi-modality.
  • Chinese-native reorganization: background knowledge, PyTorch/GPU basics, and domestic model cases (Qwen, DeepSeek, Qwen-VL) are added.
  • Each assignment provides runnable implementations with explanations of the design decisions.
  • Supports CPU debugging and MacBook MPS adaptation.
  • Includes Chinese benchmarks like C-EVAL and evaluation tools (lm-evaluation-harness, evalscope).
  • Offline-readable PDF lecture notes are available.

Use cases

  • Learners who want to understand and implement LLM components from scratch.
  • Developers seeking a structured Chinese resource for pre-training, fine-tuning, alignment, and evaluation.
  • Researchers or engineers who need practical code examples for tokenizer, Transformer, MoE, distributed training, and inference optimization.

Getting started

bashgit clone https://github.com/datawhalechina/diy-llm.gitcd diy-llm# Install basic dependencies according to specific assignment requirements

License and pricing

The license is not stated in the provided metadata. The project is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/datawhalechinadiy-llm.svg)](https://yunodb.com/tools/datawhalechinadiy-llm?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/datawhalechinadiy-llm?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/datawhalechinadiy-llm.svg" alt="Featured on YunoDB" height="20"></a>