rlhf-book
Textbook on reinforcement learning from human feedback
What is rlhf-book?
Textbook on reinforcement learning from human feedback
Overview of rlhf-book
What it is
RLHF Book is a comprehensive guide to Reinforcement Learning from Human Feedback and a broad introduction to post-training language models. The repository contains the book source, reference implementations of RLHF algorithms, diagram source files, and teaching materials. It is authored by Nathan Lambert and is available to read online, as a printed book from Manning Publications or Amazon, and as a community resource.
Key features
- Book source files in Markdown, with build scripts for HTML and PDF output
- Reference implementations for RLHF algorithms in the
code/directory, including instruction tuning, policy gradient methods (PPO, REINFORCE, GRPO, RLOO), reward model training, direct alignment methods (DPO and variants), rejection sampling, and on-policy distillation (SDPO) - Diagram source files in
diagrams/with Python generation scripts, LaTeX/TikZ sources, and YAML specifications, designed to be reusable for presentations and learning materials - Teaching materials in
teach/including courses and slides - Community translations maintained by readers in separate repositories under the same CC-BY-NC-SA license
Use cases
The book is intended for people who want to learn about reinforcement learning from human feedback and post-training of language models. The code and diagrams can be used for studying, teaching, and building upon the methods described in the book. The repository serves as a home base for learners and a reference for practitioners working at the frontier of open models.
Getting started
To build the book locally from source:bashmake html # Build HTML sitemake pdf # Build PDF (requires LaTeX)See book/README.md for detailed build instructions. For the code library, see code/README.md for setup and usage.
License and pricing
The book chapters (book/chapters) are licensed under CC-BY-NC-SA-4.0. Everything else (code/, diagrams/, scripts/, etc.) is licensed under MIT. Some images in book/images/ are unlicensed photos or screenshots. The repository is open source. The printed book is available for purchase from Manning Publications and Amazon.
這是什麼
RLHF Book 是一本關於從人類回饋中進行強化學習的綜合指南,並廣泛介紹語言模型的後期訓練。此儲存庫包含書籍原始碼、RLHF 演算法的參考實作、圖表原始檔以及教學材料。作者為 Nathan Lambert,可線上閱讀,也可從 Manning Publications 或 Amazon 購買實體書,並作為社群資源提供。
主要功能
- 書籍原始檔為 Markdown 格式,附有 HTML 和 PDF 輸出的建置腳本
code/目錄中包含 RLHF 演算法的參考實作,包括指令微調、策略梯度方法(PPO、REINFORCE、GRPO、RLOO)、獎勵模型訓練、直接對齊方法(DPO 及其變體)、拒絕取樣和同策略蒸餾(SDPO)diagrams/目錄中包含圖表原始檔,附有 Python 生成腳本、LaTeX/TikZ 原始檔和 YAML 規格,設計為可重複用於簡報和學習材料teach/目錄中包含教學材料,包括課程和投影片- 由讀者在獨立儲存庫中維護的社群翻譯,採用相同的 CC-BY-NC-SA 授權
使用情境
本書適合想學習從人類回饋中進行強化學習和語言模型後期訓練的人。程式碼和圖表可用於學習、教學以及基於書中所述方法進行建置。此儲存庫作為學習者的基地和在前沿開放模型領域工作的從業人員的參考資源。
快速開始
若要從原始檔在本機建置書籍:bashmake html # 建置 HTML 網站make pdf # 建置 PDF(需要 LaTeX)詳細建置說明請參閱 book/README.md。程式碼庫的設定與使用請參閱 code/README.md。
授權與價格
書籍章節(book/chapters)採用 CC-BY-NC-SA-4.0 授權。其他所有內容(code/、diagrams/、scripts/ 等)採用 MIT 授權。book/images/ 中的部分圖片是未經授權的照片或螢幕截圖。此儲存庫為開源。實體書可從 Manning Publications 和 Amazon 購買。
Based on the project's public README and repository metadata.