bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
What is bitsandbytes?
Accessible large language models via k-bit quantization for PyTorch.
Overview of bitsandbytes
What it is
bitsandbytes is a library that enables accessible large language models via k-bit quantization for PyTorch. It provides tools to dramatically reduce memory consumption for both inference and training of large language models.
Key features
- 8-bit optimizers use block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.
- LLM.int8() (8-bit quantization) enables large language model inference with only half the required memory and without performance degradation, using vector-wise quantization and separate outlier handling.
- QLoRA (4-bit quantization) enables large language model training by quantizing a model to 4-bits and inserting a small set of trainable low-rank adaptation (LoRA) weights.
- Includes quantization primitives for 8-bit and 4-bit operations via
bitsandbytes.nn.Linear8bitLtandbitsandbytes.nn.Linear4bit. - Includes 8-bit optimizers through the
bitsandbytes.optimmodule.
Use cases
- Reducing memory consumption for inference of large language models.
- Reducing memory consumption for training large language models.
- Enabling large language model training on consumer-grade hardware.
- Fine-tuning large language models with limited GPU memory.
Getting started
See the repository README for installation instructions and system requirements.
License and pricing
bitsandbytes is MIT licensed. It is open source and free to use.
Based on the project's public README and repository metadata.