Coalesce Deployments and Refresh with GitLab

GitLab is a popular open-source web-based DevOps platform that provides a comprehensive set of tools for software development, version control, continuous integration/continuous delivery (CI/CD), and project management. It was created by GitLab Inc. and is written in Ruby.

In this tutorial, we go through how to securely set up Coalesce Deployments and Refresh using GitLab.

Before You Begin

Access to a GitLab Account and Project. Sign up for free here.
GitLab must be set up and configured in your Coalesce Org. To use GitLab Pipelines, you must use GitLab as your git repository.
Understanding of Environments.
Understanding of the Coalesce CLI.

Setup Your Coalesce Environment

In order to use GitLab for CI/CD, one will need to set up their Coalesce project repository using GitLab. Instructions on how to do so can be found in Set Up Version Control.

You'll need an Environment in Coalesce to deploy into. Go to your main workspace and go to the Build Settings -> Environments -> Create Environment.

In the newly created Environment. Fill out the User Credentials, Storage Mappings, and Snowflake Account Identifier.

Make sure to save the configured environment and commit the changes to the GitLab.
Once the Environment is set up, you'll have to create a CLI configuration file to hold authentication credentials for orchestrating Coalesce deployments/refreshes and save it locally. It is recommended to use a service account rather than an individual user to orchestrate the deployments/refreshes.

Creating Deployment Pipelines in GitLab

In GitLab, Click Projects and navigate to your Project/Coalesce repo where you want to build your CI/CD pipeline.
Navigate to Settings -> CI/CD. Expand Secure Files.
Click Upload File and upload the secure CLI config file that you saved locally in the fourth step of Setup Your Coalesce Environment.
Navigate to Build -> Pipeline Editor.
Ensure that the pipeline that you are trying to build “lives” in the branch you are trying to configure an automated deployment for. You can also edit the pipeline config to reflect the trigger branch for the pipeline you want to deploy.

Copy/Paste the pipeline template below:

image: node:20.0.0


stages:          # List of stages for jobs, and their order of execution
- download-coa
- download-secure-file
- print-version
- coa-plan
- coa-deploy


variables:
COA_VERSION: latest # Modify this value to reflect your CLI configuration file
CONFIG_FILE_NAME: coa_config_gitlab_cicd.txt # Modify this value to reflect your CLI configuration file
COA_PROFILE: default # Modify this value to reflect your CLI configuration file


cache:
paths:
- node_modules/


download-coa:   # This job downloads the Coalesce CLI.
stage: download-coa
script:
- npm install @coalescesoftware/coa@$COA_VERSION
only:
- main


download-secure-file:       # This job downloads the secure file.
stage: download-secure-file
variables:
SECURE_FILES_DOWNLOAD_PATH: 'node_modules/demo-file-folder'
script:
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
- ls -lah node_modules/demo-file-folder/
only:
- main


print-version:   # This job prints the coa cli version.
stage: print-version
script:
- npm coa --version
only:
- main


coa-plan:   # This job creates the deployment plan.
stage: coa-plan
script:
- npx coa plan --config  /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --profile $COA_PROFILE --out /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/coa-plan.json --debug
only:
- main


coa-deploy:   # This deploys the commit in the branch based on the configuration of the coa_plan.json created in the coa-plan step.
stage: coa-deploy
script:
- npx coa deploy --config /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --plan /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/coa-plan.json --debug
only:
- main

In the above code, the three variables COA_VERSION, CONFIG_FILE_NAME, and COA_PROFILE define the CLI version, the name of the configuration file, and the profile as configured in the configuration file that you want to use to generate the deployment plan. They have been given default values of latest, coa_config_gitlab_cicd.txt, and default but you may modify these to reflect the name of your configuration file, the version of the CLI that you want to install, and the profile corresponding with the environment that you want to deploy into.
Once this pipeline is saved and committed, any successful merge or commit to the main branch of this Coalesce will result in the execution of this pipeline, and subsequent deployment into the specified Coalesce environment.
This is a simple deployment pipeline, for more advanced configuration options, please see our CLI documentation.

Using GitLab Pipelines to Schedule Refreshes

You can also schedule refreshes with GitLab pipelines using a similar script to the one above. In addition, GitLab has a built-in scheduler that you can use to run the refresh at a cadence of your choosing.

Create a new pipeline using the previous section, Creating Deployment Pipelines in GitLab. Except in Step 6, copy the following code to set up your refresh pipeline.

image: node:20.0.0




stages:          # List of stages for jobs, and their order of execution
- download-coa
- download-secure-file
- print-version
- coa-refresh




variables:
COA_VERSION: latest
CONFIG_FILE_NAME: coa_config_gitlab_cicd.txt
COA_PROFILE: default
JOBID:




cache:
paths:
- node_modules/




download-coa:   # This job downloads the Coalesce CLI.
stage: download-coa
script:
  - npm install @coalescesoftware/coa@$COA_VERSION
only:
  - main




download-secure-file:       # This job downloads the secure file.
stage: download-secure-file
variables:
  SECURE_FILES_DOWNLOAD_PATH: 'node_modules/demo-file-folder'
script:
  - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
  - ls -lah node_modules/demo-file-folder/




print-version:   # This job prints the coa cli version.
stage: print-version
script:
  - npm coa --version




coa-refresh: # This refreshes the specified environment.
 stage: coa-refresh
 script:
   - npx coa refresh --config /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --profile $COA_PROFILE --jobID $JOBID

In the above code, the three variables COA_VERSION, CONFIG_FILE_NAME, and COA_PROFILE define the CLI version, the name of the configuration file, and the profile as configured in the configuration file that you want to use to generate the deployment plan.

They have been given default values of latest, coa_config_gitlab_cicd.txt, and default but you may modify these to reflect the name of your configuration file, the version of the CLI that you want to install, and the profile corresponding with the environment that you want to deploy into.

The optional jobID variable specifying the ID of the job that you want to refresh in the environment specified. If the $JOBID parameter is removed, all the deployed nodes in the Environment will be refreshed.
If you want to refresh multiple jobs sequentially, you will have to add additional stages that copy the coa-refresh stage and specify a different$JOB_IDfor each one. For additional configuration options, please refer to the CLI documentation.
Once your pipeline is ready, you can set a schedule by navigating to Pipeline Schedules -> Create a New Pipeline Schedule and set the desired cadence for your pipeline.
Click Create pipeline schedule and verify that the schedule and the pipeline branch is set properly in the next screen.