server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
What is server?
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Overview of server
What it is
Triton Inference Server is an open source inference serving software that streamlines AI inferencing. It enables teams to deploy any AI model from multiple deep learning and machine learning frameworks, including TensorRT, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. It supports inference across cloud, data center, edge and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia.
Key features
- Supports multiple deep learning frameworks
- Supports multiple machine learning frameworks
- Concurrent model execution
- Dynamic batching
- Sequence batching and implicit state management for stateful models
- Provides Backend API that allows adding custom backends and pre/post processing operations
- Supports writing custom backends in Python (Python-based backends)
- Model pipelines using Ensembling or Business Logic Scripting (BLS)
- HTTP/REST and GRPC inference protocols based on the KServe protocol
- A C API and Java API allow Triton to link directly into your application for edge and other in-process use cases
- Metrics indicating GPU utilization, server throughput, server latency, and more
Use cases
Triton Inference Server is used for deploying AI models in production environments across cloud, data center, edge, and embedded devices. It supports real-time inference, batched inference, ensembles, and audio/video streaming. It is part of NVIDIA AI Enterprise, a software platform that accelerates the data science pipeline and streamlines the development and deployment of production AI.
Getting started
bash# Step 1: Create the example model repositorygit clone -b r26.10 https://github.com/triton-inference-server/server.gitcd server/docs/examples./fetch_models.sh# Step 2: Launch triton from the NGC Triton containerdocker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.10-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx# Step 3: Sending an Inference Requestdocker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.10-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg
License and pricing
This project is licensed under the BSD-3-Clause license. It is open source and free to use.
Based on the project's public README and repository metadata.