vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

What is vllm-ascend?

Community maintained hardware plugin for vLLM on Huawei Ascend

Overview of vllm-ascend

What it is

vLLM Ascend (vllm-ascend) is a community maintained hardware plugin for running vLLM seamlessly on the Huawei Ascend NPU. It is the recommended approach for supporting the Ascend backend within the vLLM community. It adheres to the principles outlined in the RFC: Hardware pluggable, providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM. By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Experts (MoE), Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU.

Key features

  • Hardware pluggable interface that decouples Ascend NPU integration with vLLM.
  • Supports Transformer-like, Mixture-of-Experts (MoE), Embedding, and Multi-modal LLMs.
  • Community maintained with weekly meetings and active contribution.
  • Multiple maintained branches for different vLLM versions (main, releases/v0.13.0, releases/v0.18.0, releases/v0.23.0, releases/v0.30.0, rfc/).
  • CI commitment for vLLM main branch and specific version tags.
  • Official documentation and support matrix for supported models and features.

Use cases

Running vLLM on Huawei Ascend NPU hardware for inference and deployment of large language models. Suitable for users who need to leverage Ascend NPU computing resources for development, testing, or production workloads. Can be used in fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios as demonstrated by user stories with LLaMA-Factory, verl, TRL, and GPUStack.

Getting started

See the repository README for detailed installation and quick start instructions. The recommended versions are v0.30.0 (latest stable) and v0.23.0 (stable). Prerequisites include: Atlas 800I A2 series, Atlas 800I A3 series, Atlas 300I Duo, 950PR&950DT Products hardware; Linux OS; Python == 3.12; CANN == 9.2.1; PyTorch == 2.10.0, TorchNPU == 2.10.0.post7; and vLLM (same version as vllm-ascend).

License and pricing

Licensed under Apache License 2.0. The project is open source with no stated pricing.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/vllm-projectvllm-ascend.svg)](https://yunodb.com/tools/vllm-projectvllm-ascend?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/vllm-projectvllm-ascend?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/vllm-projectvllm-ascend.svg" alt="Featured on YunoDB" height="20"></a>