BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
What is BentoML?
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Overview of BentoML
What it is
BentoML is a Python library for building online serving systems optimized for AI apps and model inference. It allows users to turn any model inference script into a REST API server with standard Python type hints. The library supports any ML framework, modality, and inference runtime.
Key features
- Easily build APIs for any AI/ML model with just a few lines of code.
- Docker containers made simple: manage environments, dependencies, and model versions with a config file; automatically generates Docker images.
- Maximize CPU/GPU utilization with built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline, and multi-model inference-graph orchestration.
- Fully customizable: implement custom APIs or task queues with custom business logic, model inference, and multi-model composition.
- Ready for production: develop, run, and debug locally; deploy to production with Docker containers or Kubernetes.
Use cases
BentoML is used for building model inference APIs, job queues, LLM apps, multi-model pipelines, and more. It supports a wide range of models including LLMs (e.g., Llama 3.2, Mistral), image generation (e.g., Stable Diffusion 3 Medium), embeddings, audio, computer vision, and advanced examples like function calling, LangGraph, and CrewAI.
Getting started
Install BentoML:# Requires Python≥3.9pip install -U bentomlDefine APIs in a service.py file. Run locally:pip install torch transformers # additional dependencies for local runbentoml serveDeploy using Docker:bentoml buildbentoml containerize summarization:latestdocker run --rm -p 3000:3000 summarization:latest
License and pricing
License: Apache License 2.0. The project is open source.
Based on the project's public README and repository metadata.