vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
What is vllm-ascend?
Community maintained hardware plugin for vLLM on Huawei Ascend
Overview of vllm-ascend
What it is
vLLM Ascend (vllm-ascend) is a community maintained hardware plugin for running vLLM seamlessly on the Huawei Ascend NPU. It is the recommended approach for supporting the Ascend backend within the vLLM community. It adheres to the principles outlined in the RFC: Hardware pluggable, providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM. By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Experts (MoE), Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU.
Key features
- Hardware pluggable interface that decouples Ascend NPU integration with vLLM.
- Supports Transformer-like, Mixture-of-Experts (MoE), Embedding, and Multi-modal LLMs.
- Community maintained with weekly meetings and active contribution.
- Multiple maintained branches for different vLLM versions (main, releases/v0.13.0, releases/v0.18.0, releases/v0.23.0, releases/v0.30.0, rfc/).
- CI commitment for vLLM main branch and specific version tags.
- Official documentation and support matrix for supported models and features.
Use cases
Running vLLM on Huawei Ascend NPU hardware for inference and deployment of large language models. Suitable for users who need to leverage Ascend NPU computing resources for development, testing, or production workloads. Can be used in fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios as demonstrated by user stories with LLaMA-Factory, verl, TRL, and GPUStack.
Getting started
See the repository README for detailed installation and quick start instructions. The recommended versions are v0.30.0 (latest stable) and v0.23.0 (stable). Prerequisites include: Atlas 800I A2 series, Atlas 800I A3 series, Atlas 300I Duo, 950PR&950DT Products hardware; Linux OS; Python == 3.12; CANN == 9.2.1; PyTorch == 2.10.0, TorchNPU == 2.10.0.post7; and vLLM (same version as vllm-ascend).
License and pricing
Licensed under Apache License 2.0. The project is open source with no stated pricing.
Based on the project's public README and repository metadata.