HRM-Text

HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.

What is HRM-Text?

HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.

Overview of HRM-Text

What it is

HRM-Text is a 1B parameter text generation model based on the HRM (Hierarchical Recurrent) architecture. It is strengthened by task completion and latent space reasoning. The repository provides a full pretraining framework, making foundation model pretraining accessible with 130-600x less compute and 150-900x less data. It uses PrefixLM sequence packing, FlashAttention 3 kernels, PyTorch FSDP2 training, evaluation, and checkpoint conversion tooling. The project is hosted at https://github.com/sapientinc/HRM-Text.

Key features

  • 1B parameter text generation model based on the HRM architecture.
  • Full pretraining framework with 130-600x less compute and 150-900x less data compared to typical approaches.
  • Supports PrefixLM sequence packing, FlashAttention 3, and PyTorch FSDP2.
  • Includes evaluation scripts and checkpoint conversion to Hugging Face Transformers format.
  • Provides fine-tuning (SFT) support for instruction data.

Use cases

  • Pretraining a foundation text generation model from scratch with limited compute resources (estimated ~$1000 for the L size).
  • Fine-tuning a pretrained HRM-Text checkpoint on custom instruction-following data.
  • Evaluating model performance on benchmarks such as GSM8k, MATH, DROP, MMLU, ARC-C, HellaSwag, Winogrande, and BoolQ.

Getting started

See the repository README for full instructions. The recommended setup uses Docker: bash docker run --gpus all --ipc=host --network=host -it -v $PWD:/workspace sapientai/hrm-text:latest For pretraining the L size on one 8xH100 node: bash OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 torchrun --nproc_per_node=8 pretrain.py arch/size@arch=L lr=2.5e-4 global_batch_size=172032

License and pricing

The license is Apache-2.0. The model is open source. Pricing is not stated; the README estimates cloud GPU costs (e.g., ~$800 for L size, ~$1472 for XL size based on $2/H100 hour).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/sapientinchrm-text.svg)](https://yunodb.com/tools/sapientinchrm-text?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/sapientinchrm-text?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/sapientinchrm-text.svg" alt="Featured on YunoDB" height="20"></a>