Refreshing an Environment
Refresh is responsible for running the data transformations defined in your data warehouse metadata. This typically involves DML (Data Manipulation Language) SQL statements such as MERGE, INSERT, UPDATE, and TRUNCATE which will perform transformations on the actual data. Use refresh when you want to update your pipeline with any new changes from your data warehouse.
Deploy Before Refreshing
You can only refresh if you've deployed your pipeline.
Before You Begin
You'll need to get an authentication token. Review the steps in Connecting to the API.
API
Refresh jobs can be triggered with the Start Job endpoint. Trigger Job to Run. Below is an example Job.
curl --request POST \
--url https://app.coalescesoftware.io/scheduler/startRun \
--header 'Authorization: Bearer <YOUR-TOKEN> ' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '
{
"runDetails": {
"parallelism": 16,
"environmentID": "4",
"excludeNodesSelector": "{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM } OR { location: SAMPLE name: NATION } OR { location: SAMPLE name: ORDERS } OR { location: SAMPLE name: PART } OR { location: SAMPLE name: PARTSUPP } OR { location: SAMPLE name: REGION } OR { location: SAMPLE name: SUPPLIER } OR { location: QA name: STG_PARTSUPP } OR { location: PROD name: STG_PARTSUPP }",
"jobID": "4"
},
"userCredentials": {
"snowflakeAuthType": "Basic",
"snowflakeRole": "ACCOUNTADMIN",
"snowflakeWarehouse": "COMPUTE_WH",
"snowflakeUsername": "SOMEUSER",
"snowflakePassword": "SOMEPASS"
}
}
'
CLI
Refresh jobs can be triggered via our CLI tool coa
using coa refresh
. Learn more in our CLI Commands documentation.
Jobs
To refresh a subset of data, you can create a Job that will run at each Refresh. Learn how to create Jobs.
Updated 29 days ago