bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

What is bitsandbytes?

Accessible large language models via k-bit quantization for PyTorch.

Overview of bitsandbytes

What it is

bitsandbytes is a library that enables accessible large language models via k-bit quantization for PyTorch. It provides tools to dramatically reduce memory consumption for both inference and training of large language models.

Key features

  • 8-bit optimizers use block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.
  • LLM.int8() (8-bit quantization) enables large language model inference with only half the required memory and without performance degradation, using vector-wise quantization and separate outlier handling.
  • QLoRA (4-bit quantization) enables large language model training by quantizing a model to 4-bits and inserting a small set of trainable low-rank adaptation (LoRA) weights.
  • Includes quantization primitives for 8-bit and 4-bit operations via bitsandbytes.nn.Linear8bitLt and bitsandbytes.nn.Linear4bit.
  • Includes 8-bit optimizers through the bitsandbytes.optim module.

Use cases

  • Reducing memory consumption for inference of large language models.
  • Reducing memory consumption for training large language models.
  • Enabling large language model training on consumer-grade hardware.
  • Fine-tuning large language models with limited GPU memory.

Getting started

See the repository README for installation instructions and system requirements.

License and pricing

bitsandbytes is MIT licensed. It is open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/bitsandbytes-foundationbitsandbytes.svg)](https://yunodb.com/tools/bitsandbytes-foundationbitsandbytes?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/bitsandbytes-foundationbitsandbytes?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/bitsandbytes-foundationbitsandbytes.svg" alt="Featured on YunoDB" height="20"></a>