flash-linear-attention
🚀 Efficient implementations for emerging model architectures
What is flash-linear-attention?
🚀 Efficient implementations for emerging model architectures
Overview of flash-linear-attention
What it is
Flash Linear Attention is a software library that provides hardware-efficient building blocks, training-ready layers, and components for modern sequence models. It covers linear attention, sparse attention, state space models, and hybrid LLM architectures. The implementations are platform-agnostic and have been verified on NVIDIA, AMD, and Intel hardware.
Key features
- Includes implementations for a wide range of models such as RetNet, GLA, DeltaNet, Mamba2, Gated Slot Attention, and many others.
- Provides fused modules for efficient training, including a fused linear and cross-entropy layer.
- Supports variable-length inputs and offers a flexible way to train hybrid models.
- Includes kernels for attention mechanisms like FlashMoBA, FlashQLA, and TileLang backend support.
- Offers context parallel support for distributed training across the sequence dimension.
- Includes implementations for emerging architectures like Gated DeltaNet 2, Kimi Delta Attention, and Stick-Breaking Attention.
Use cases
Flash Linear Attention is designed for researchers and developers working on sequence modeling tasks who need efficient implementations of modern attention and state space model architectures. It is suitable for training large language models, developing hybrid architectures, and experimenting with novel attention mechanisms. The library can be used for both research and production environments where hardware efficiency is critical.
Getting started
See the repository README for installation and usage instructions.
License and pricing
The project is licensed under the MIT License and is open source.
Based on the project's public README and repository metadata.