dagster
An orchestration platform for the development, production, and observation of data assets.
What is dagster?
An orchestration platform for the development, production, and observation of data assets.
Overview of dagster
What it is
Dagster is an orchestration platform for the development, production, and observation of data assets. It is designed for developing and maintaining data assets such as tables, data sets, machine learning models, and reports. Users declare data assets as Python functions, and Dagster helps run them at the right time to keep assets up-to-date. It is cloud-native and supports the whole development lifecycle from local development to production.
Key features
- Declarative programming model: define data assets as Python functions.
- Integrated lineage and observability: built-in metadata cataloging, diagnostics, and lineage tracking.
- Best-in-class testability: supports unit tests, integration tests, and CI/CD best practices.
- Robust multi-tenant, multi-tool orchestration engine that scales technically and organizationally.
- Unified control plane: centralize metadata with observability, diagnostics, cataloging, and lineage.
- Growing library of integrations with popular data tools.
Use cases
- Developing and maintaining data assets like tables, datasets, ML models, and reports.
- Running pipelines in production with a robust orchestration engine.
- Managing data complexity with centralized metadata and observability.
- Implementing CI/CD practices for data pipelines.
Getting started
Dagster is available on PyPI and supports Python 3.9 through 3.14. Install with:
uv add dagster dagster-webserver dagster-dg-cli
For a full tutorial, see the repository README.
License and pricing
Dagster is licensed under Apache 2.0 and is open source.
Based on the project's public README and repository metadata.