Skip to main content

Creating and Run Jobs

Before You Begin​

You'll need to get an authentication token to run refreshes. Tokens are used for the Trigger Job to Run API or using refresh with the CLI. Review the steps in Connecting to the API.

Deploy Before Refreshing

You can only refresh if you've deployed your pipeline.

Step 1: Create a Job​

  1. Go to Jobs in the Build sidebar.
  2. Select the + sign to create a new Job.
  3. Select Edit to add selector queries. Selector queries allow you to select a subset of Nodes to be refreshed.
  4. Take note of your Job ID. In this example it's jobID: 3. You'll need the Job ID to run it as part of the refresh.
Creating a job from the Mapping Grid
Add Subgraphs to Jobs

You can add a subgraph to a Job by using a selector query or by drag and drop.

Step 2: Commit Your Job​

Jobs need to be committed into Git and deployed to an environment before they can be used. You can read more about making commits in our Git Integration article.

Example of a job committed to Git

Step 3: Configure Your Environment​

Go to Build Settings > Environments and check that the environment you want to refresh is configured. It should have:

Step 4: Deploy Your Pipeline​

Jobs can only be run if they have been deployed first. Review our Deployment Overview to learn different ways to deploy your pipeline.

Step 5: Run Your Jobs​

Use the Coalesce Scheduler (fastest), API or CLI to run a Job.

Scheduler​

The built-in Coalesce Scheduler provides the easiest and fastest way to automate your environment refresh.

API​

Jobs can be triggered with the Start Job endpoint. Trigger Job to Run. By only passing the environmentID and leaving the jobID out, you can refresh the entire environment. You can also use the excludeNodesSelector and includeNodesSelector to override the Jobs created. To avoid setting the selectors manually each run, we recommend using Jobs to save and manage nodes.

curl --request POST \
     --url https://app.coalescesoftware.io/scheduler/startRun \
     --header 'Authorization: Bearer YOUR-API-TOKEN' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "runDetails": {
    "parallelism": 16,
    "jobID": "4",
    "environmentID": "10"
  },
  "userCredentials": {
  # Use the credentials for your platform. Review the API documentation for more authentication examples.

    "snowflakeAuthType": "Basic",
    "snowflakeRole": "ACCOUNTADMIN",
    "snowflakeWarehouse": "COMPUTE_WH",
    "snowflakeUsername": "SOMEUSER",
    "snowflakePassword": "SOMEPASS"

    "platformKind": "databricks",
    "databricksClientSecret": "******",
    "databricksClientID": "*****",
    "databricksPath": "sql warehouse path",
    "databricksAuthType": "OAuthM2M"
  }
}
'

CLI​

Refresh jobs can be triggered using our CLI tool coa using coa refresh. Learn more in our CLI Commands documentation. By only passing the environmentID and leaving the jobID out, you can refresh the entire environment. You can also use the excludeNodesSelector and includeNodesSelector to override the Jobs created. To avoid setting the selectors manually each run, we recommend using Jobs to save and manage nodes.

CLI Set Up

Make sure to setup your CLI. Review CLI set up.

This example assumes you are using a coa config file.

  coa refresh --environmentID 1  --jobID 4

Coalesce App​

Only existing, deployed Jobs can be run from the Coalesce App.

Go to the Deploy page, and next to each deploy, click on the menu, and select Run Job.

The image shows a user interface for managing deployment environments and jobs. The Dev Env section indicates the status of the environment and its deploy history, with a Deploy button and a series of colored indicators representing deployment events. Below, there is a Test Refresh Job section, where a dropdown menu allows the user to run the job, as indicated by the arrow and the label Run Job.