airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

What is airflow?

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Overview of airflow

What it is

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative. Airflow uses DAGs to orchestrate tasks; the scheduler executes tasks on an array of workers while following specified dependencies.

Key features

  • Workflows are defined in code, enabling dynamic DAG generation and parameterization.
  • The scheduler executes tasks on workers while following specified dependencies.
  • Rich command line utilities support operating on DAGs.
  • A rich user interface helps visualize pipelines running in production, monitor progress, and troubleshoot issues.
  • The framework includes a wide range of built-in operators and can be extended.
  • Airflow uses the Jinja templating engine for customization.
  • Tasks can pass metadata using XCom.

Use cases

Airflow works best with workflows that are mostly static and slowly changing. It is commonly used to process data. It is also used to orchestrate machine learning workflows, including training, retraining, evaluation, and deployment. It is increasingly used to orchestrate agentic and LLM-based workloads, coordinating AI pipeline steps such as data prep, tool calls, model invocation, and evaluation rather than acting as the agent itself. Airflow is not a streaming solution, but it is often used to process real-time data by pulling data off streams in batches.

Getting started

The README directs users to the official Airflow website documentation for installing Airflow, getting started, and a more complete tutorial. It does not include install or quick-start commands. See the repository README and official documentation.

License and pricing

Apache-2.0. Metadata lists the pricing type as open_source. No other pricing is stated.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/apacheairflow.svg)](https://yunodb.com/tools/apacheairflow?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/apacheairflow?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/apacheairflow.svg" alt="Featured on YunoDB" height="20"></a>