coalesce.ioTake documentation survey

Schedule Jobs With Github Actions and the API

Suggest Edits

GitHub Actions serves as a comprehensive platform for continuous integration and continuous delivery (CI/CD), enabling the automation of your build, testing, and deployment processes. It empowers you to design workflows that automatically build and test each pull request submitted to your repository, and seamlessly deploy successfully merged pull requests to the production environment.

In this how-to, we will walk you through best practices for scheduling Coalesce refreshes through the Github actions interface to allow you to automate the triggering of your coalesce jobs to be on a schedule or upon approved pull request into your main branch in accordance with CI/CD best practices.

Before You Begin

Set up a Coalesce Environment (Optional)

It is essential that your Coalesce workspace is synced with a Github repo in order to utilize Github Actions.

  1. Create a new environment to use GitHub Actions with. It's recommended to use a environment just for GitHub Actions.
  2. Setup Key-Pair Authentication on the environment you created.
  3. Deploy the new environment to branch you want.
  4. Click Generate Access Token. You will need it to authenticate the API call made from Github Actions.

Set up Github Variables and Environment

  1. In GitHub, go to repo you are using with Coalesce.
  2. Click Settings > Security > Secrets and variable > Actions.
  3. You'll create two repository secrets by clicking the secrets tab, called MY_COALESCE_SECRET, and SF_PRIVATE_KEY.
    1. MY_COALESCE_SECRET - Your API access token from your environment.
    2. SF_PRIVATE_KEY - Snowflake Private Key from Snowflake Key-Pair Authentication.
  4. Add your private key. The private key must contain explicitly typed newline (\n) characters at the end of each line in the .pem file when saved as plain text in the private key secret in GitHub.

Private key example

-----BEGIN PRIVATE KEY-----\n

MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCv2xVhFpaM6hhf\n

ce4U5GRfdArGkoqkL2EBRs0zGMn1YYfQ8+zDuN9YkMTNC1pNxQptGn921teGk0wv\n

cMP+I83P390jqXh56TlQtwn2reRXH7OlLdELttof4VGYb4I6KpdBhDaid8bys2FE\n

f0r948EXM81Euh9FgmMbc4KzeF1tBDyU0sqAcAJCQXOl95jUR6Wqdp04LXJVoGmI\n

-----END PRIVATE KEY-----

Update Repo With Secrets

  1. Create a new folder called .github in your main branch.
  2. Create a subfolder within .github called workflows and file within workflows called coalesce_test.yml.
  3. Copy and paste the following sample code into coalesce_test.yml.
  4. Update the "environmentID": "3" and "jobID": "2" to match your Coalesce environment.

In the default state, this workflow is triggered under three circumstances:

  • A pull request into the main branch
  • A successful merging of an open pull request into the main branch
  • Execution on the cron scheduler (commented out). If you want to modify the CRON syntax, you can use any posix cron scheduler.

The file also includes workflow_dispatch so you can trigger the job manually or turn it on and off in the GitHub app.

# This is a basic workflow to help you get started with triggering Coalesce Jobs with 

# Github Actions.




name: Coalesce Trigger.




# Controls when the workflow will run.

on:

  # Triggers the workflow on push or pull request events but only for the "main" branch.

  push:

    branches: [ "main" ]

  pull_request:

    branches: [ "main" ]

  # Add Schedule in Cron format if desired.

  # schedule:

  # - cron: "*/15 * * * *"




  # Allows you to run this workflow manually from the Actions tab if needed.

  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel.

jobs:

  # This workflow contains a single job called "Trigger Coalesce Job".

  Trigger_Coalesce_Job:

    # The type of runner that the job will run on.

    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job.

    steps:

      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it

      - uses: actions/checkout@v3

      # Runs a single command using the runners shell

      - name: Run an initialization script.

        run: echo "Run Starting"

      # Runs a set of commands using the runners shell

      - name: Run Coalesce Job.

        env: # Obtain env secrets.

          sf_private_key_: ${{ secrets.SF_PRIVATE_KEY }}

          coalesce_token_: ${{ secrets.MY_COALESCE_SECRET }}

        run: |

          curl --location 'https://app.coalescesoftware.io/scheduler/startRun' \

          --header 'accept: application/json' \

          --header 'content-type: application/json' \

          --header 'Authorization: Bearer ${{ secrets.MY_COALESCE_SECRET }}' \

          --data-raw '{
            "runDetails": {
                "parallelism": 16,
                "environmentID": "3",
                "jobID": "2"
            },
            "userCredentials": {
                "snowflakeAuthType": "KeyPair",
                "snowflakeKeyPairKey": "${{ secrets.SF_PRIVATE_KEY }}"
            }
        }'

Resources


Updated 10 days ago