train-llm-from-scratch
A straightforward method for training your LLM, from downloading data to generating text.
What is train-llm-from-scratch?
A straightforward method for training your LLM, from downloading data to generating text.
Overview of train-llm-from-scratch
What it is
A method for training a large language model (LLM) from scratch using PyTorch, based on the paper Attention is All You Need. The project covers the entire pipeline from downloading raw text to generating text, including pretraining and post-training stages such as supervised fine-tuning (SFT), reward model training, PPO, DPO, and GRPO. All algorithms are implemented in plain PyTorch without external libraries like trl, peft, or transformers. The repository provides scripts that can train a billion-parameter model on a single GPU or a small model on a laptop CPU.
Key features
- Implements a transformer model from scratch using PyTorch, including both the original architecture and a modern version with rotary positions, RMSNorm, SwiGLU, grouped-query and latent attention, Mixture of Experts, and a KV cache.
- Hand-written training algorithms: Muon optimizer, LoRA, IPO, SimPO, Dr. GRPO, DAPO, GSPO, speculative decoding, and int8 weight quantization.
- Covers the full training pipeline: raw text → tokens → transformer → next-token loss → base model → SFT → reward model → {PPO, DPO} → GRPO → evaluation and chat.
- Supports training on a single GPU or on a laptop CPU (for small models).
- Provides two config systems: a simple Python constants file for legacy pretraining, and JSON-based configs with command-line overrides for all other stages.
- Includes scripts for data preparation, training, text generation, evaluation, and a Streamlit control panel.
Use cases
- Students learning how transformers work and how to implement them from scratch.
- Developers who want to train a custom LLM on their own data without relying on high-level libraries.
- Researchers experimenting with post-training techniques such as DPO, PPO, and GRPO on small transformers.
- Anyone who wants to train a small language model on a laptop for educational or prototyping purposes.
Getting started
Prerequisites: basic understanding of object-oriented programming, neural networks, and PyTorch. Clone the repository and install in editable mode: bash git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git cd train-llm-from-scratch pip install -e . Optional extras: pip install -e .[train] for datasets and wandb, pip install -e .[ui] for Streamlit, pip install -e .[docs] for documentation, pip install -e .[all] for everything. To train a small model on a laptop CPU (no GPU needed): bash python scripts/prepare_tiny_data.py python scripts/train_transformer.py --preset tiny --arch modern python scripts/generate_text.py --model_path models/tiny.pt
License and pricing
The project is licensed under the MIT License and is open source (free).
Based on the project's public README and repository metadata.