PowerInfer
High-speed Large Language Model Serving for Local Deployment
What is PowerInfer?
High-speed Large Language Model Serving for Local Deployment
Overview of PowerInfer
What it is
PowerInfer is a high-speed Large Language Model (LLM) inference engine designed for local deployment on a personal computer equipped with a single consumer-grade GPU. It exploits the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. Hot-activated neurons are preloaded onto the GPU, while cold-activated neurons are computed on the CPU, reducing GPU memory demands and CPU-GPU data transfers.
Key features
- Locality-centric design: Utilizes sparse activation and 'hot'/'cold' neuron concept for efficient LLM inference.
- Hybrid CPU/GPU utilization: Seamlessly integrates memory/computation capabilities of CPU and GPU for balanced workload.
- Easy integration: Compatible with popular ReLU-sparse models.
- Local deployment ease: Designed and deeply optimized for local deployment on consumer-grade hardware.
- Backward compatibility: Supports inference with llama.cpp's model weights for compatibility.
Use cases
Local deployment of large language models on consumer-grade hardware, enabling low-latency inference and serving on a single GPU. Suitable for running models like Falcon-40B, Llama2 family, ProSparse Llama2 family, and Bamboo-7B on a personal computer.
Getting started
git clone https://github.com/Tiiny-AI/PowerInfer
cd PowerInfer
pip install -r requirements.txt
### Build with CMake (example for NVIDIA GPU):
cmake -S . -B build -DLLAMA_CUBLAS=ON
cmake --build build --config Release
For AMD GPU or CPU-only builds, see the repository README.
License and pricing
License: MIT. Open source.
Based on the project's public README and repository metadata.