HRM-Text
HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.
What is HRM-Text?
HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.
Overview of HRM-Text
What it is
HRM-Text is a 1B parameter text generation model based on the HRM (Hierarchical Recurrent) architecture. It is strengthened by task completion and latent space reasoning. The repository provides a full pretraining framework, making foundation model pretraining accessible with 130-600x less compute and 150-900x less data. It uses PrefixLM sequence packing, FlashAttention 3 kernels, PyTorch FSDP2 training, evaluation, and checkpoint conversion tooling. The project is hosted at https://github.com/sapientinc/HRM-Text.
Key features
- 1B parameter text generation model based on the HRM architecture.
- Full pretraining framework with 130-600x less compute and 150-900x less data compared to typical approaches.
- Supports PrefixLM sequence packing, FlashAttention 3, and PyTorch FSDP2.
- Includes evaluation scripts and checkpoint conversion to Hugging Face Transformers format.
- Provides fine-tuning (SFT) support for instruction data.
Use cases
- Pretraining a foundation text generation model from scratch with limited compute resources (estimated ~$1000 for the L size).
- Fine-tuning a pretrained HRM-Text checkpoint on custom instruction-following data.
- Evaluating model performance on benchmarks such as GSM8k, MATH, DROP, MMLU, ARC-C, HellaSwag, Winogrande, and BoolQ.
Getting started
See the repository README for full instructions. The recommended setup uses Docker: bash docker run --gpus all --ipc=host --network=host -it -v $PWD:/workspace sapientai/hrm-text:latest For pretraining the L size on one 8xH100 node: bash OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 torchrun --nproc_per_node=8 pretrain.py arch/size@arch=L lr=2.5e-4 global_batch_size=172032
License and pricing
The license is Apache-2.0. The model is open source. Pricing is not stated; the README estimates cloud GPU costs (e.g., ~$800 for L size, ~$1472 for XL size based on $2/H100 hour).
Based on the project's public README and repository metadata.