Skip to main content

Airflow setup

info

This feature is in Beta mode, please reach out if you want to activate it on your instance.

Catalog currently integrates through Marquez and OpenLineage.

Configure Airflow​

  • If using Airflow 2.3 to 2.6 : add openlineage-airflow to the requirements / packages to install within the Airflow instance
  • If using Airflow 2.7 : nothing to install
  • Add the following environment variables:
    • OPENLINEAGE_URL : the URL of the HTTP backend
    • OPENLINEAGE_API_KEY : the API key to allow Airflow to write to Marquez
    • OPENLINEAGE_NAMESPACE : the namespace associated to the Airflow instance

Credentials​

You must provide Catalog with following elements:

  • Marquez API URL and API key (OPENLINEAGE_URL, OPENLINEAGE_API_KEY)
  • namespace configured for Airflow (OPENLINEAGE_NAMESPACE)
  • namespace for the main warehouse source (set by OpenLineage)
info

If you do not have your own Marquez server, please reach out to catalog-support@coalesce.io