maestro

Maestro: Netflix’s Workflow Orchestrator

What is maestro?

Maestro: Netflix’s Workflow Orchestrator

Overview of maestro

What it is

Maestro is a general-purpose workflow orchestrator developed by Netflix. It provides a fully managed workflow-as-a-service (WAAS) to data platform users. It serves thousands of users including data scientists, data engineers, machine learning engineers, software engineers, content producers, and business analysts. Maestro schedules hundreds of thousands of workflows and millions of jobs every day, operating with strict service level objectives (SLOs) even during traffic spikes.

Key features

  • Fully managed workflow-as-a-service (WAAS) for data platform users - Schedules hundreds of thousands of workflows and millions of jobs daily - Operates with strict SLOs under high traffic - Highly scalable and extensible to support new use cases - Supports Kubernetes for running workflow steps - Provides a Python SDK client (maestro-sdk) for programmatic workflow creation and management - Includes job templates and foreach step flattening via extensions

Use cases

Maestro is used for orchestrating data and machine learning workflows at Netflix. It supports a wide range of users and use cases, including data pipeline scheduling, ML model training pipelines, and business analytics workflows. Its extensibility allows it to adapt to new requirements.

Getting started

Prerequisites: Git, Java 21, Gradle, Docker. Build and run the server: ./gradlew build ./gradlew bootRun Install the Python SDK: pip install maestro-sdk For detailed instructions on creating and managing workflows via REST API or Python SDK, see the repository README.

License and pricing

Maestro is licensed under the Apache License, Version 2.0. It is open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/netflixmaestro.svg)](https://yunodb.com/tools/netflixmaestro?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/netflixmaestro?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/netflixmaestro.svg" alt="Featured on YunoDB" height="20"></a>