chitu

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

What is chitu?

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

Overview of chitu

What it is

Chitu「赤兔」is a high-performance inference framework for large language models, focusing on efficiency, flexibility, and availability. It is designed as a production-grade large model inference engine, addressing the progressive needs of enterprise AI deployment from small-scale experiments to large-scale deployments. The project is hosted on GitHub under the repository thu-pacman/chitu.

Key features

  • Supports multiple hardware platforms: NVIDIA (from flagship to older series), domestic chips including Huawei Ascend 910B, Moore Threads GPU, Muxi, Haiguang, and others.
  • Scalable across scenarios: from pure CPU deployment, single GPU deployment to large-scale cluster deployment.
  • Long-term stable operation: suitable for production environments, capable of handling concurrent business traffic.
  • Supports FP4 online conversion to FP8/BF16 efficient operator implementations (e.g., for DeepSeek-R1 671B FP4 quantized version).
  • Supports CPU+GPU heterogeneous hybrid inference, enabling single-card inference of DeepSeek-R1 671B.
  • Provides a single-file executable chitu.run for launching complex tasks such as multi-node, multi-instance, and PD separation.
  • Supports models including DeepSeek, Qwen, GLM, Kimi, and others.

Use cases

Chitu is intended for enterprise AI deployment scenarios that require high performance and stability. Use cases include deploying large language models in production environments, running inference on diverse hardware (NVIDIA, domestic chips), scaling from a single GPU to a multi-node cluster, and performing hybrid CPU+GPU inference for memory-constrained models like DeepSeek-R1 671B.

Getting started

It is recommended to use the chitu.run executable for deployment. Download it from the Assets page in the Releases section of the repository. For complete installation and usage instructions, refer to the development manual in the repository README.

License and pricing

This project is licensed under the Apache License v2.0. See the LICENSE file for details. The codebase also includes code snippets from other open-source projects (with SPDX license annotations in the code) and third-party submodules under other open-source licenses, located in the LICENSES/ and third_party/ directories respectively. The pricing type is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/thu-pacmanchitu.svg)](https://yunodb.com/tools/thu-pacmanchitu?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/thu-pacmanchitu?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/thu-pacmanchitu.svg" alt="Featured on YunoDB" height="20"></a>