airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
What is airflow?
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Overview of airflow
What it is
Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative. Airflow uses DAGs to orchestrate tasks; the scheduler executes tasks on an array of workers while following specified dependencies.
Key features
- Workflows are defined in code, enabling dynamic DAG generation and parameterization.
- The scheduler executes tasks on workers while following specified dependencies.
- Rich command line utilities support operating on DAGs.
- A rich user interface helps visualize pipelines running in production, monitor progress, and troubleshoot issues.
- The framework includes a wide range of built-in operators and can be extended.
- Airflow uses the Jinja templating engine for customization.
- Tasks can pass metadata using XCom.
Use cases
Airflow works best with workflows that are mostly static and slowly changing. It is commonly used to process data. It is also used to orchestrate machine learning workflows, including training, retraining, evaluation, and deployment. It is increasingly used to orchestrate agentic and LLM-based workloads, coordinating AI pipeline steps such as data prep, tool calls, model invocation, and evaluation rather than acting as the agent itself. Airflow is not a streaming solution, but it is often used to process real-time data by pulling data off streams in batches.
Getting started
The README directs users to the official Airflow website documentation for installing Airflow, getting started, and a more complete tutorial. It does not include install or quick-start commands. See the repository README and official documentation.
License and pricing
Apache-2.0. Metadata lists the pricing type as open_source. No other pricing is stated.
Based on the project's public README and repository metadata.