PowerInfer

High-speed Large Language Model Serving for Local Deployment

What is PowerInfer?

High-speed Large Language Model Serving for Local Deployment

Overview of PowerInfer

What it is

PowerInfer is a high-speed Large Language Model (LLM) inference engine designed for local deployment on a personal computer equipped with a single consumer-grade GPU. It exploits the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. Hot-activated neurons are preloaded onto the GPU, while cold-activated neurons are computed on the CPU, reducing GPU memory demands and CPU-GPU data transfers.

Key features

  • Locality-centric design: Utilizes sparse activation and 'hot'/'cold' neuron concept for efficient LLM inference.
  • Hybrid CPU/GPU utilization: Seamlessly integrates memory/computation capabilities of CPU and GPU for balanced workload.
  • Easy integration: Compatible with popular ReLU-sparse models.
  • Local deployment ease: Designed and deeply optimized for local deployment on consumer-grade hardware.
  • Backward compatibility: Supports inference with llama.cpp's model weights for compatibility.

Use cases

Local deployment of large language models on consumer-grade hardware, enabling low-latency inference and serving on a single GPU. Suitable for running models like Falcon-40B, Llama2 family, ProSparse Llama2 family, and Bamboo-7B on a personal computer.

Getting started

git clone https://github.com/Tiiny-AI/PowerInfer
cd PowerInfer
pip install -r requirements.txt
### Build with CMake (example for NVIDIA GPU):
cmake -S . -B build -DLLAMA_CUBLAS=ON
cmake --build build --config Release

For AMD GPU or CPU-only builds, see the repository README.

License and pricing

License: MIT. Open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/tiiny-aipowerinfer.svg)](https://yunodb.com/tools/tiiny-aipowerinfer?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/tiiny-aipowerinfer?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/tiiny-aipowerinfer.svg" alt="Featured on YunoDB" height="20"></a>