FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
What is FastDeploy?
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Overview of FastDeploy
What it is
FastDeploy is a high-performance inference and deployment toolkit for large language models (LLMs) and vision-language models (VLMs) based on PaddlePaddle. It provides production-ready deployment solutions with features such as load-balanced PD decomposition, unified KV cache transfer, OpenAI API service compatibility with vLLM, full quantization format support, advanced acceleration techniques, and multi-hardware support.
Key features
- Load-balanced PD decomposition with context caching and dynamic instance role switching - Unified KV cache transfer library supporting intelligent NVLink/RDMA selection - OpenAI API service and vLLM compatibility for single-command deployment - Full quantization format support: W8A16, W8A8, W4A16, W4A8, W2A16, FP8 - Advanced acceleration: speculative decoding, multi-token prediction (MTP), and chunked prefill - Multi-hardware support: NVIDIA GPU, Kunlunxin XPU, Hygon DCU, Iluvatar GPU, Enflame GCU, Muxi GPU, Intel Gaudi
Use cases
FastDeploy is suitable for deploying LLMs and VLMs in production environments requiring high throughput and low latency. It can be used for offline inference, online serving, and scenarios that benefit from advanced techniques like PD separation, speculative decoding, prefix caching, and load balancing.
Getting started
See the repository README for installation instructions.
License and pricing
FastDeploy is licensed under Apache-2.0 and is open source.
Based on the project's public README and repository metadata.