Troubleshooting Deploys and Refreshes
You can view the results of a deploy or refresh three ways:
- The Deploy Page
- API
- CLI
The Deploy Page
In the Deploy Page you check the status of current or completed runs.
The feed includes:
- Connection details
- Parameters used
- Details for each Node + steps for a Node
- Snowflake query ID
- Error messages
- The run status next to the Environment name.
- Open each Environment to see the status of deploy and refreshes. They are color coded. You can hover over each one to get a quick overview of the run information.
- The type of run and the status is color coded.
- A list of all the runs, the type, and status.
The color will depend on the status:
- Green - It was successful with no errors.
- Yellow - The Refresh was successful, but had column or Node test failures.
- Red - The Deploy or Refresh failed.
Individual Run Page
Clicking on any of the Deploy or Refresh status will take you to the individual run page.
Understanding the Individual Run Page
In this example, you'll review a failed deployment, but this can also be applied to Refresh and Jobs.
The heading contains:
- The phase and number of nodes that failed in that phase. In this example it's one node in the add phase.
Run failure reason: RunPhasedDeploy encountered failures: delete phase - none; alter phase - none; add phase - 1 - The status and number of nodes. In this example, 11 were successful and one failed.
The row contains:
- The failed node. The failed node in this example is,
V_Nation.
Double clicking on the error will open a new window that includes the error message and the SQL for that stage.
The error for this run is SQL compilation error: Object 'DOCS_TESTING.DEV.STG_NATION1' does not exist or not authorized.
Once the error is resolved, you can run the Job from the point of failure, rollback your deployment, or update your pipeline and redeploy.
Deploy and Refresh Status
After a deploy or refresh, you'll be able to see which Nodes were run and their status.
| Status | Description |
|---|---|
| Successful | The node finished executing all stages successfully. |
| Waiting | The node is waiting to run. |
| Queued | The node is part of the current run and is waiting execution. |
| Running | The node is currently running and stages may have already been completed. |
| Failed | The node has finished running and encountered an error. Some stages might have completed successfully. |
| Skipped | A step was skipped. Some stages might have completed successfully. |
| Canceled | The run was canceled while running. Some stages might have completed successfully. |
API Run Status
Use the List Run Results API to get the results of a deploy or refresh.
In this example, you'll review a failed deployment, but this can also be applied to Refresh and Jobs.
{
"nodeID": "97378833-0000-46bc-9582-e68ded3ae016",
"runState": "error",
"isRunning": false,
"name": "STG_ORDERS",
"queryResults": [
{
"endTime": "2024-07-16T15:13:19.357Z",
"error": {
"errorString": "SQL compilation error:\nDatabase 'DOCUMENTATION_SNOWFLAKE_TEST' does not exist or not authorized.",
"errorDetail": "002003"
},
"isRunning": false,
"name": "Delete Table",
"queryID": "01b5b591-0905-106b-0012-7b030fac7362",
"sql": "\n DROP TABLE IF EXISTS \"DOCUMENTATION_SNOWFLAKE_TEST\".\"DEV\".\"STG_ORDERS_COALESCE_INTERNAL_TABLE\"",
"startTime": "2024-07-16T15:13:10.641Z",
"status": "Failure",
"success": false
}
]
},
...
The response is organized by nodeID. The response will contain the current status, the node name, the name of the stage that was executed, and the error message if applicable. Review the full response example for all fields.
You can see "nodeID": "97378833-0000-46bc-9582-e68ded3ae016", failed because "SQL compilation error:\nDatabase 'DOCUMENTATION_SNOWFLAKE_TEST' does not exist or not authorized.". The database, it was trying to run the stage, "name": "Delete Table" from does not exist.
Once the error is resolved, you can run the Job from the point of failure, rollback your deployment, or update your pipeline and redeploy.
CLI Run Status
Use coa runs to get the deployment status.
When deploying or refreshing using the CLI, add the --out flag to return in depth results as JSON. For example, coa deploy --out debug.json.
In this example, you'll review a failed deployment, but this can also be applied to Refresh and Jobs.
....
|2024-07-18T17:00:45.155Z|[RENDERER]()|)|info:Pyodide initialization complete in 1.3305933750000003 seconds
|2024-07-18T17:00:45.159Z|[RENDERER]()|)|info:Pyodide initialization complete in 1.3246505830000004 seconds
|2024-07-18T17:00:45.198Z|[RENDERER]()|)|info:Pyodide initialization complete in 1.363098707999999 seconds
|2024-07-18T17:00:55.135Z|[CLI]|(org:alRPfzuHZouUUN0Zupc9)|(env:2)|(user:Owb7B45m0UNSL4OumkDNpcFifTA3)|run:28)()|)|error:An error occurred during Deployment: {
error: {
errorString: 'RunPhasedDeploy encountered failures: \n' +
' delete phase - 7; \n' +
' alter phase - none; \n' +
' add phase - 1\n' +
' ',
errorDetail: 'RunPhasedDeploy encountered failures: \n' +
' delete phase - e3e488cc-232f-4e7f-a266-16fe8b4dfd79,e38d8549-bfd0-4044-80ef-108ab297b431,e1e2ea88-de6e-4930-8aaa-dd5ffd1851d9,a7ee70b6-d2a3-4a59-acea-f5a0951b6981,97378833-0000-46bc-9582-e68ded3ae016,3a85b97f-6e6e-4d26-8986-8161f1dd378c,7771b0de-037f-4d9a-a1ad-22400fc0414b; \n' +
' alter phase - none; \n' +
' add phase - bde6322a-45d0-4964-9606-6a0d0a9867d5\n' +
' '
}
}
When a deploy finished it will return a status in the command line.
The errorString will tell you which nodes failed in each phase.
- The delete phase has 7 nodes encounter errors.
- The alter phase had no errors.
- The add phase had 1 node encounter errors.
The errorDetail will return a list of node IDs that failed for each phase.
Using JSON Output
By adding --out to return the results as JSON, you can get a detailed error message. Here you can see the error details: SQL compilation error:\nObject 'DOCUMENTATION_DOCS_TESTING.DEV.STG_LINEITEM' does not exist or not authorized."
{
"runResults": {
"runStartTime": {
"seconds": 1721322042,
"nanoseconds": 300000000
},
"runEndTime": {
"seconds": 1721322054,
"nanoseconds": 238000000
},
"runType": "deploy",
"runStatus": "failed",
"runID": 28,
"runResults": [
{
"nodeID": "bde6322a-45d0-4964-9606-6a0d0a9867d5",
"queryResultSequence": {
"isRunning": false,
"name": "STG_LINEITEM",
"queryResults": [
{
"exportedRefs": [],
"sql": "\nSELECT 1 FROM \"DOCUMENTATION_DOCS_TESTING\".\"DEV\".\"STG_LINEITEM\" LIMIT 0",
"renderEndTime": {
"seconds": 1721322044,
"nanoseconds": 129000000
},
"type": "sql",
"fields": [],
"startTime": {
"seconds": 1721322043,
"nanoseconds": 719000000
},
"success": false,
"error": {
"errorDetail": "002003",
"errorString": "SQL compilation error:\nObject 'DOCUMENTATION_DOCS_TESTING.DEV.STG_LINEITEM' does not exist or not authorized."
},
"rows": [],
"stageExecutionStartTime": {
"seconds": 1721322051,
"nanoseconds": 791000000
},
"invalidExportedRefs": [],
"warehouse": null,
"queryID": "01b5c13c-0905-144e-0012-7b030fbc081a",
"isRunning": false,
"renderStartTime": {
"seconds": 1721322043,
"nanoseconds": 719000000
},
"stageExecutionEndTime": {
"seconds": 1721322053,
"nanoseconds": 565000000
},
"endTime": {
"seconds": 1721322053,
"nanoseconds": 565000000
},
"name": "Validating Source Exists",
"status": "Failure"
}
]
},
"runState": "error"
},
Once the error is resolved, you can run the Job from the point of failure, rollback your deployment, or update your pipeline and redeploy.
Node and Column Test Failures
This feature is currently available in a private preview and may not be accessible to all customers.
If a Node or column test failures during Refresh, then the Refresh status will be yellow and the test failure will be in the results. The API will also have a hasTestFailure:true added.
API
The following endpoints will have a hasTestFailure:true added:
In the following example the field hasTestFailure:true is there because a test failed. It failed on the test for "name": "N_REGIONKEY: Unique". You can look at error and see what the failure was.
"error": {
"errorString": "Test failed - records matching fail condition were returned",
"errorDetail": "Test failed - records matching fail condition were returned"
},
The status will be success, while success will be false. This combination also indicates that while the refresh run finished, there was a failure and in this example, it was the column level test for unique.
"status": "Success",
"success": false,
{
"data": [
{
"nodeID": "596874d4-5555-4114-9bdc-b2de778fd98c",
"runState": "complete",
"isRunning": false,
"name": "NATION",
"queryResults": [
{
"endTime": "2025-03-26T15:44:27.156Z",
"isRunning": false,
"name": "Validating Source Exists",
"queryID": "01bb44d0-0907-dbe8-0012-7b031a5930e6",
"sql": "\nSELECT 1 FROM \"SNOWFLAKE_SAMPLE_DATA\".\"TPCH_SF100\".\"NATION\" LIMIT 0",
"startTime": "2025-03-26T15:44:26.920Z",
"status": "Success",
"success": true,
"warehouse": "COMPUTE_WH"
}
]
},
{
"nodeID": "99293300-f747-4990-a176-e82d2973694e",
"runState": "complete",
"hasTestFailures": true,
"isRunning": false,
"name": "STG_NATION",
"queryResults": [
{
"endTime": "2025-03-26T15:44:28.397Z",
"isRunning": false,
"name": "Truncate Stage Table",
"queryID": "01bb44d0-0907-d01f-0012-7b031a590a5e",
"sql": "\n\t\t\tTRUNCATE IF EXISTS \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\"\n\t\n\t\t\n\t\n\t\n\t\t\n\t\n\t\n\t\t\n\t\n\t\t\t\n\t\t\t\t\n",
"startTime": "2025-03-26T15:44:27.968Z",
"status": "Success",
"success": true,
"warehouse": "COMPUTE_WH"
},
{
"endTime": "2025-03-26T15:44:29.597Z",
"isRunning": false,
"name": "Insert STG_NATION",
"queryID": "01bb44d0-0907-dadd-0012-7b031a58fbee",
"rowsInserted": 25,
"sql": "\n\t\n\t\t\t\tINSERT INTO \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\"\n\t\t\t\t(\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_NATIONKEY\",\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_NAME\",\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_REGIONKEY\",\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_COMMENT\"\n\t\t\t\t\t\n\t\t\t\t)\n\t\t\t\n\t\n\t\t\tSELECT\n\t\t\t\n \"NATION\".\"N_NATIONKEY\" AS \"N_NATIONKEY\", \n\t\t\t\n \"NATION\".\"N_NAME\" AS \"N_NAME\", \n\t\t\t\n \"NATION\".\"N_REGIONKEY\" AS \"N_REGIONKEY\", \n\t\t\t\n \"NATION\".\"N_COMMENT\" AS \"N_COMMENT\"\n\t\t\t\n\t\n\t\t\tFROM \"SNOWFLAKE_SAMPLE_DATA\".\"TPCH_SF100\".\"NATION\" \"NATION\"\n\t\n\t\t\t\n\t\n\t\t\n\t\n\t\n\n\n\n\t\n\n\t\n\t\t\n\t\t\t\n",
"startTime": "2025-03-26T15:44:28.956Z",
"status": "Success",
"success": true,
"warehouse": "COMPUTE_WH"
},
{
"endTime": "2025-03-26T15:44:30.739Z",
"isRunning": false,
"name": "N_NATIONKEY: Unique",
"queryID": "01bb44d0-0907-dc64-0012-7b031a58cc06",
"sql": "SELECT 1 WHERE EXISTS (\n\t\t\t\n SELECT \"N_NATIONKEY\", COUNT(*)\n FROM \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\" \n GROUP BY \"N_NATIONKEY\"\n HAVING COUNT(*) > 1\n\t\t\n\t\n\t\t\n\t\n\t\t\n\t\t\t\n)",
"startTime": "2025-03-26T15:44:30.327Z",
"status": "Success",
"success": true,
"warehouse": "COMPUTE_WH"
},
{
"endTime": "2025-03-26T15:44:31.783Z",
"error": {
"errorString": "Test failed - records matching fail condition were returned",
"errorDetail": "Test failed - records matching fail condition were returned"
},
"isRunning": false,
"name": "N_REGIONKEY: Unique",
"queryID": "01bb44d0-0907-dbe8-0012-7b031a5930fa",
"sql": "SELECT 1 WHERE EXISTS (\n\t\t\t\n SELECT \"N_REGIONKEY\", COUNT(*)\n FROM \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\" \n GROUP BY \"N_REGIONKEY\"\n HAVING COUNT(*) > 1\n\t\t\n\t\n\t\t\n\t)",
"startTime": "2025-03-26T15:44:31.330Z",
"status": "Success",
"success": false,
"warehouse": "COMPUTE_WH"
}
]
}
]
}
Retry From Failure
If a Job failed you can retry the Job from the point of failure. You can use the:
- API - Use the Rerun a Job endpoint.
- Coalesce App - You can select the Job that failed and click Retry From Failure.
- Scheduler - You can configure scheduled Jobs to retry from beginning or from the point of failure.
When you retry a Job from failure, in the Coalesce app, you'll be able to see the previous run details.
Rollback Your Deployment
You can redeploy the commit that was deployed just prior to the deployment you wish to rollback. Learn more in Rollback a Deployment.
Redeploy
You can edit your pipeline, commit the changes, and redeploy to fix any errors.