FastDeploy

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

What is FastDeploy?

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

Overview of FastDeploy

What it is

FastDeploy is a high-performance inference and deployment toolkit for large language models (LLMs) and vision-language models (VLMs) based on PaddlePaddle. It provides production-ready deployment solutions with features such as load-balanced PD decomposition, unified KV cache transfer, OpenAI API service compatibility with vLLM, full quantization format support, advanced acceleration techniques, and multi-hardware support.

Key features

  • Load-balanced PD decomposition with context caching and dynamic instance role switching - Unified KV cache transfer library supporting intelligent NVLink/RDMA selection - OpenAI API service and vLLM compatibility for single-command deployment - Full quantization format support: W8A16, W8A8, W4A16, W4A8, W2A16, FP8 - Advanced acceleration: speculative decoding, multi-token prediction (MTP), and chunked prefill - Multi-hardware support: NVIDIA GPU, Kunlunxin XPU, Hygon DCU, Iluvatar GPU, Enflame GCU, Muxi GPU, Intel Gaudi

Use cases

FastDeploy is suitable for deploying LLMs and VLMs in production environments requiring high throughput and low latency. It can be used for offline inference, online serving, and scenarios that benefit from advanced techniques like PD separation, speculative decoding, prefix caching, and load balancing.

Getting started

See the repository README for installation instructions.

License and pricing

FastDeploy is licensed under Apache-2.0 and is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/paddlepaddlefastdeploy.svg)](https://yunodb.com/tools/paddlepaddlefastdeploy?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/paddlepaddlefastdeploy?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/paddlepaddlefastdeploy.svg" alt="Featured on YunoDB" height="20"></a>