serving
A flexible, high-performance serving system for machine learning models
What is serving?
A flexible, high-performance serving system for machine learning models
Overview of serving
What it is
TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. It handles the inference aspect of machine learning, taking models after training and managing their lifetimes. It provides clients with versioned access via a high-performance, reference-counted lookup table. TensorFlow Serving offers out-of-the-box integration with TensorFlow models but can be extended to serve other types of models and data.
Key features
- Can serve multiple models or multiple versions of the same model simultaneously - Exposes both gRPC and HTTP inference endpoints - Allows deployment of new model versions without changing any client code - Supports canarying new versions and A/B testing experimental models - Adds minimal latency to inference time due to efficient, low-overhead implementation - Features a scheduler that groups individual inference requests into batches for joint execution on GPU, with configurable latency controls - Supports many servables: TensorFlow models, embeddings, vocabularies, feature transformations, and even non-TensorFlow-based machine learning models
Use cases
TensorFlow Serving is used for serving machine learning models in production environments. It enables canarying new model versions and A/B testing experimental models. It can serve multiple models or multiple versions of the same model simultaneously, allowing gradual rollouts and experimentation without disrupting existing clients.
Getting started
The quickest way to get started is using Docker. To serve a TensorFlow model in 60 seconds, run: bash # Download the TensorFlow Serving Docker image and repo docker pull tensorflow/serving git clone https://github.com/tensorflow/serving # Location of demo models TESTDATA=$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata # Start TensorFlow Serving container and open the REST API port docker run -t --rm -p 8501:8501 -v $TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two -e MODEL_NAME=half_plus_two tensorflow/serving & # Query the model using the predict API curl -d 'instances: [1.0, 2.0, 5.0]' -X POST http://localhost:8501/v1/models/half_plus_two:predict # Returns => { predictions: [2.5, 3.0, 4.5] }
License and pricing
Licensed under Apache-2.0. Open source and free to use.
Based on the project's public README and repository metadata.