Skip to main content

Airflow setup

Learn how to integrate Catalog with Apache Airflow using Marquez and OpenLineage for lineage tracking.

Beta Feature

This feature is in Beta. Reach out if you want to activate it on your instance.

Catalog integrates with Marquez and OpenLineage to capture lineage from your Airflow DAGs.

Configure Airflow​

To connect Airflow to Catalog, install the OpenLineage integration and set the required environment variables.

  • If using Airflow 2.3 through 2.6: add openlineage-airflow to the requirements or packages to install within the Airflow instance
  • If using Airflow 2.7: nothing to install
  • Add the following environment variables:
    • OPENLINEAGE_URL: the URL of the HTTP backend
    • OPENLINEAGE_API_KEY: the API key to allow Airflow to write to Marquez
    • OPENLINEAGE_NAMESPACE: the namespace associated with the Airflow instance

Credentials​

You must provide Catalog with the following elements:

  • Marquez API URL and API key: OPENLINEAGE_URL, OPENLINEAGE_API_KEY
  • Namespace configured for Airflow (OPENLINEAGE_NAMESPACE)
  • Namespace for the main warehouse source (set by OpenLineage)
No Marquez Server

If you do not have your own Marquez server, reach out to the support team.