LitServe

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

What is LitServe?

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

Overview of LitServe

What it is

LitServe is a minimal Python framework for building custom AI inference servers. It allows developers to define exactly how inference works for models, agents, RAG, or pipelines. It provides control over batching, routing, streaming, and orchestration without requiring MLOps glue or configuration files. LitServe is built on FastAPI but optimized for AI, claiming 2× faster performance with AI-specific multi-worker handling. It supports any PyTorch model and can be self-hosted or deployed on Lightning AI.

Key features

  • Custom inference logic with full control over batching, caching, and routing.
  • Support for agents, RAG, chatbots, image models, video, speech, text, and more.
  • Multi-GPU autoscaling, batching, and streaming included.
  • No MLOps glue code required; build full AI systems in one place using LitAPI.
  • Instant setup with setup() to connect models, databases, and data.
  • Deploy anywhere: self-host or one-click deploy with Lightning AI.
  • OpenAPI compliant and OpenAI compatible.

Use cases

LitServe can be used to build inference APIs, agents, chatbots, RAG systems, MCP servers, or multi-model pipelines. It is suitable for deploying any pipeline or model, including LLMs, vision models, audio models, speech models, and classical ML models. Examples include Llama 3.2, Whisper, Stable Diffusion, and random forest.

Getting started

Install LitServe via pip: bash pip install litserve Run a basic server: bash python server.py For deployment on Lightning Cloud: bash lightning deploy server.py --cloud

License and pricing

LitServe is licensed under Apache-2.0 and is open source. Pricing is not stated beyond being open source; a free tier is available for Lightning Cloud deployment.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/lightning-ailitserve.svg)](https://yunodb.com/tools/lightning-ailitserve?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/lightning-ailitserve?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/lightning-ailitserve.svg" alt="Featured on YunoDB" height="20"></a>