BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

What is BentoML?

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Overview of BentoML

What it is

BentoML is a Python library for building online serving systems optimized for AI apps and model inference. It allows users to turn any model inference script into a REST API server with standard Python type hints. The library supports any ML framework, modality, and inference runtime.

Key features

  • Easily build APIs for any AI/ML model with just a few lines of code.
  • Docker containers made simple: manage environments, dependencies, and model versions with a config file; automatically generates Docker images.
  • Maximize CPU/GPU utilization with built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline, and multi-model inference-graph orchestration.
  • Fully customizable: implement custom APIs or task queues with custom business logic, model inference, and multi-model composition.
  • Ready for production: develop, run, and debug locally; deploy to production with Docker containers or Kubernetes.

Use cases

BentoML is used for building model inference APIs, job queues, LLM apps, multi-model pipelines, and more. It supports a wide range of models including LLMs (e.g., Llama 3.2, Mistral), image generation (e.g., Stable Diffusion 3 Medium), embeddings, audio, computer vision, and advanced examples like function calling, LangGraph, and CrewAI.

Getting started

Install BentoML:# Requires Python≥3.9pip install -U bentomlDefine APIs in a service.py file. Run locally:pip install torch transformers # additional dependencies for local runbentoml serveDeploy using Docker:bentoml buildbentoml containerize summarization:latestdocker run --rm -p 3000:3000 summarization:latest

License and pricing

License: Apache License 2.0. The project is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/bentomlbentoml.svg)](https://yunodb.com/tools/bentomlbentoml?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/bentomlbentoml?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/bentomlbentoml.svg" alt="Featured on YunoDB" height="20"></a>