Skip to main content

Orchestrate Jobs With GitHub Actions and the API

GitHub Requires Node 20

Node 16 is reached the end of it's life. GitHub now requires Node 20.

Learn more in GitHub Actions: Transitioning from Node 16 to Node 20.

GitHub Actions serves as a comprehensive platform for continuous integration and continuous delivery (CI/CD), enabling the automation of your build, testing, and deployment processes. It empowers you to design workflows that automatically build and test each pull request submitted to your repository, and seamlessly deploy successfully merged pull requests to the production environment.

In this how-to, we will walk you through best practices for scheduling Coalesce refreshes through the GitHub actions interface to allow you to automate the triggering of your Coalesce jobs to be on a schedule or upon approved pull request into your main branch in accordance with CI/CD best practices.

Before You Begin​

You'll need:

  • GitHub: GitHub account with admin privileges to configure pipelines and manage secrets.
  • Project: A Project with version control configured using your chosen platform.
  • Environment: At least one Environment configured for deployment.
  • Access Token: Your Coalesce access token for the CLI configuration file.

Create and Configure an Environment​

  1. Create a new Environment for CI/CD deployments. It's recommended to use a dedicated Environment for automated deployments.
  2. Configure authentication for your Environment based on your data platform:
  3. Configure Storage Mappings for your Environment.
  4. Deploy the Environment to verify configuration.

Data Platform Permissions​

Snowflake​

  • USAGE privilege on the database and schema.
  • SELECT, INSERT, UPDATE, and DELETE privileges on tables involved in transformations.
  • EXECUTE privilege for procedures or functions referenced by Coalesce.
  • Ability to create Snowflake Key-Pair Authentication (recommended for automated deployments).

Databricks​

Set up GitHub Variables and Environment​

  1. In GitHub, go to repo you are using with Coalesce.
  2. Click Settings > Security > Secrets and variable > Actions.
  3. You'll create two repository secrets by clicking the secrets tab, called MY_COALESCE_SECRET and DATABRICKS_CLIENT_SECRET.
    1. MY_COALESCE_SECRET - Your API access token from your environment.
    2. DATABRICKS_CLIENT_SECRET - Client Secret from Databricks OAuth Machine-to-Machine setup.

Update Repo With Secrets​

  1. Create a new folder called .github in your main branch.
  2. Create a subfolder within .github called workflows and file within workflows called coalesce_test.yml.
  3. Copy and paste the following sample code into coalesce_test.yml.
  4. Update the "environmentID": "3" and "jobID": "2" to match your Coalesce environment.

In the default state, this workflow is triggered under three circumstances:

  • A pull request into the main branch
  • A successful merging of an open pull request into the main branch
  • Execution on the cron scheduler (commented out). If you want to modify the cron syntax, you can use any POSIX cron scheduler.

The file also includes workflow_dispatch so you can trigger the job manually or turn it on and off in the GitHub app.

name: Deploy to PROD

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  COA_VERSION: latest

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-node@v3
        with:
          node-version: 22.19.0

      - run: echo '${{ secrets.COA_CONFIG }}' >> COA_CONFIG

      # Install Coalesce CLI tool, if not installed already
      - run: npm list | grep "@coalescesoftware/coa@${{ env.COA_VERSION }}" || npm install @coalescesoftware/coa@${{ env.COA_VERSION }}

      # Print version number
      - run: npx coa --version

      # Create Deployment Plan
      - run: npx coa plan --config COA_CONFIG --out ./coa-plan --debug

      # Deploy Plan
      - run: npx coa deploy --config COA_CONFIG --plan ./coa-plan --debug

What's Next?​