Skip to main content

GitHub Actions for dbt Sync

Use GitHub Actions to synchronize your dbt manifest file with GCS and keep your data catalog up to date.

Create the Action​

You need to create a folder in your dbt repository with the path <repository>/.github/workflows/

Inside you need to create a YAML file, we will name ours send_to_catalog.yml

The content is :

name: Send manifest to Catalog
on: [push]
jobs:
  send-manifest-to-catalog:
    runs-on: ubuntu-latest
    steps:
      - name: Imports GitHub code source
        uses: actions/checkout@v2 
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install castor-extractor
      - name: Launch python script
        run: castor-upload -c '${{ secrets.SERVICE_ACCOUNT_JSON }}' -s <source_id> -f dbt_manifest.json -t DBT

Create the Secrets​

There's a reference guide in order to understand the manipulation of secrets using GitHub Actions.

Here is a screenshot of the minified version of the Service Account JSON content stored and passed to the Python function.

Service Account JSON stored in GitHub Secrets

Push the Code​

Once that all the code is pushed to the repository, the workflow will be triggered. Then you will see something like this in the Actions tab of the repository.

GitHub Actions workflow run in the Actions tab

The date and time of the file will be in UTC.