Skip to main content

Orchestrate Deploys and Refreshes with GitLab and the CLI

GitLab is a popular open-source web-based DevOps platform that provides a comprehensive set of tools for software development, version control, continuous integration/continuous delivery (CI/CD), and project management. It was created by GitLab Inc. and is written in Ruby.

In this tutorial, we go through how to securely set up Coalesce Deployments and Refresh using GitLab.

Before You Begin​

You'll need:

  • GitLab: GitLab account with admin privileges to configure pipelines and manage secrets.
  • Project: A Project with version control configured using your chosen platform.
  • Environment: At least one Environment configured for deployment.
  • Access Token: Your Coalesce access token for the CLI configuration file.

Create and Configure an Environment​

  1. Create a new Environment for CI/CD deployments. It's recommended to use a dedicated Environment for automated deployments.
  2. Configure authentication for your Environment based on your data platform:
  3. Configure Storage Mappings for your Environment.
  4. Deploy the Environment to verify configuration.

Data Platform Permissions​

Snowflake​

  • USAGE privilege on the database and schema.
  • SELECT, INSERT, UPDATE, and DELETE privileges on tables involved in transformations.
  • EXECUTE privilege for procedures or functions referenced by Coalesce.
  • Ability to create Snowflake Key-Pair Authentication (recommended for automated deployments).

Databricks​

Configure Your CLI​

The Coalesce CLI orchestrates your deployments and refreshes within CI/CD pipelines. You'll create a configuration file with credentials required to authenticate the CLI.

  1. Follow the instructions in CLI Setup to install the CLI.
  2. Create a new text file, for example, coa_config.txt, and save it securely. This file contains the authentication credentials your pipeline will use.

Your configuration file should include profiles for each Environment.

[production]
token=your_coalesce_token
domain=https://app.coalesce.io
platformKind=Databricks
databricksAuthType=OAuthM2M
databricksClientID=client_id_here
databricksClientSecret=client_secret_here
databricksAccountHost=https://accounts.cloud.databricks.com
databricksWorkspaceHost=https://your-workspace.cloud.databricks.com
databricksPath=/sql/1.0/warehouses/abc123xyz
environmentID=456

Configuration Notes:

  • Create separate profiles using [profile_name] for different Environments.
  • Use service accounts or service principals rather than individual user accounts.
  • For Snowflake, include the private key path when using key-pair authentication.
  • Store this file securely - it contains sensitive credentials.

Creating Deployment Pipelines in GitLab​

  1. Within GitLab, Click Projects and navigate to your Project/Coalesce repo where you want to build your CI/CD pipeline.

    The image displays a dark-themed GitLab interface with a navigation menu on the left, highlighting the Projects section. The main section advertises a free trial of GitLab.com Ultimate and shows a list of projects under the Projects heading. One project named Coalesce / GitLab_ci_cd is highlighted with a red border, indicating the owner of the project.

  2. Navigate to Settings > CI/CD. Expand Secure Files.

    The image displays a GitLab project interface with a dark theme, focusing on the GitLab_ci_cd project. On the left side, the navigation menu includes various sections such as Learn GitLab, Pinned, Issues, and Merge requests, with Settings highlighted at the bottom. The right side shows a dropdown menu under Settings, highlighting the CI/CD option, indicating the current focus on continuous integration and deployment settings.

  3. Click Upload File and upload the secure CLI config file that you saved locally.

    The image shows a section titled Secure Files in a GitLab interface with a dark theme. It explains that Secure Files can be used to store files needed by pipelines, such as Android keystores or Apple provisioning profiles. The section lists one file named coa_config_gitlab_cicd.txt, which was uploaded 21 hours ago, with options to upload a new file or delete the existing one.

  4. Navigate to Build > Pipeline Editor.

    The image shows a dark-themed GitLab interface focusing on the project navigation menu. The menu includes sections such as Issues, Merge requests, Manage, Plan, Code, Build, Pipelines, Jobs, and Pipeline editor, with the Pipeline editor section highlighted. Other sections like Pipeline schedules, Artifacts, Secure, Deploy, Operate, Monitor, Analyze, and Settings are also visible below the highlighted section.

  5. Ensure that the pipeline that you are trying to build “lives” in the branch you are trying to configure an automated deployment for. You can also edit the pipeline config to reflect the trigger branch for the pipeline you want to deploy.

  6. Copy and paste the pipeline template below:

    image: node:20.0.0

    stages:          # List of stages for jobs, and their order of execution
    - download-coa
    - download-secure-file
    - print-version
    - coa-plan
    - coa-deploy

    variables:
    COA_VERSION: latest # Modify this value to reflect your CLI configuration file
    CONFIG_FILE_NAME: coa_config_gitlab_cicd.txt # Modify this value to reflect your CLI configuration file
    COA_PROFILE: default # Modify this value to reflect your CLI configuration file

    cache:
    paths:
        - node_modules/

    download-coa:    # This job downloads the Coalesce CLI.
    stage: download-coa
    script:
        - npm install @coalescesoftware/coa@$COA_VERSION
    only:
        - main

    download-secure-file:       # This job downloads the secure file.
    stage: download-secure-file
    variables:
        SECURE_FILES_DOWNLOAD_PATH: 'node_modules/demo-file-folder'
    script:
        - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
        - ls -lah node_modules/demo-file-folder/
    only:
        - main

    print-version:   # This job prints the coa cli version.
    stage: print-version
    script:
        - npm coa --version
    only:
        - main

    coa-plan:        # This job creates the deployment plan.
    stage: coa-plan
    script:
        - npx coa plan --config  /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --profile $COA_PROFILE --out /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/coa-plan.json --debug
    only:
        - main

    coa-deploy:      # This deploys the commit in the branch based on the configuration of the coa_plan.json created in the coa-plan step.
    stage: coa-deploy
    script:
        - npx coa deploy --config /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --plan /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/coa-plan.json --debug
    only:
        - main

  7. In the above code, the three variables COA_VERSION, CONFIG_FILE_NAME, and COA_PROFILE define the CLI version, the name of the configuration file, and the profile as configured in the configuration file that you want to use to generate the deployment plan. They have been given default values of latest, coa_config_gitlab_cicd.txt, and default but you may modify these to reflect the name of your configuration file, the version of the CLI that you want to install, and the profile corresponding with the environment that you want to deploy into.

  8. Once this pipeline is saved and committed, any successful merge or commit to the main branch of this Coalesce will result in the execution of this pipeline, and subsequent deployment into the specified Coalesce environment.

  9. This is a simple deployment pipeline, for more advanced configuration options, please see our CLI documentation.

Using GitLab Pipelines to Schedule Refreshes​

You can also schedule refreshes with GitLab pipelines using a similar script to the one above. In addition, GitLab has a built-in scheduler that you can use to run the refresh at a cadence of your choosing.

  1. Create a new pipeline using the previous section, Creating Deployment Pipelines in GitLab. Except in Step 6, copy the following code to set up your refresh pipeline.

    image: node:20.0.0

    stages:          # List of stages for jobs, and their order of execution
    - download-coa
    - download-secure-file
    - print-version
    - coa-refresh

    variables:
    COA_VERSION: latest
    CONFIG_FILE_NAME: coa_config_gitlab_cicd.txt
    COA_PROFILE: default
    JOBID:

    cache:
    paths:
        - node_modules/

    download-coa:    # This job downloads the Coalesce CLI.
    stage: download-coa
    script:
        - npm install @coalescesoftware/coa@$COA_VERSION
    only:
        - main

    download-secure-file:       # This job downloads the secure file.
    stage: download-secure-file
    variables:
        SECURE_FILES_DOWNLOAD_PATH: 'node_modules/demo-file-folder'
    script:
        - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
        - ls -lah node_modules/demo-file-folder/

    print-version:   # This job prints the coa cli version.
    stage: print-version
    script:
        - npm coa --version

    coa-refresh:     # This refreshes the specified environment.
    stage: coa-refresh
    script:
        - npx coa refresh --config /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --profile $COA_PROFILE --jobID $JOBID

  2. In the above code, the three variables COA_VERSION, CONFIG_FILE_NAME, and COA_PROFILE define the CLI version, the name of the configuration file, and the profile as configured in the configuration file that you want to use to generate the deployment plan.

    They have been given default values of latest, coa_config_gitlab_cicd.txt, and default but you may modify these to reflect the name of your configuration file, the version of the CLI that you want to install, and the profile corresponding with the environment that you want to deploy into.

    The optional jobID variable specifying the ID of the job that you want to refresh in the environment specified. If the $JOBID parameter is removed, all the deployed nodes in the Environment will be refreshed.

  3. If you want to refresh multiple jobs sequentially, you will have to add additional stages that copy the coa-refresh stage and specify a different $JOB_IDfor each one. For additional configuration options, please refer to the CLI documentation.

  4. Once your pipeline is ready, you can set a schedule by navigating to Pipeline Schedules > Create a New Pipeline Schedule and set the desired cadence for your pipeline.

    The image shows a dark-themed GitLab interface focusing on the project navigation menu. The menu includes sections such as Issues, Merge requests, Manage, Plan, Code, Build, Pipelines, Jobs, and Pipeline editor, with the Pipeline schedules section highlighted. Other sections like Artifacts, Secure, Deploy, Operate, Monitor, Analyze, and Settings are also visible below the highlighted section. The image shows a dark-themed GitLab interface for scheduling a new pipeline. The form includes fields for the description, interval pattern with options for daily, weekly, monthly, and custom schedules, and a Cron syntax input field. Additional fields allow the user to select the Cron timezone, target branch or tag, and input variables, with options to activate the schedule and buttons to create the pipeline schedule or cancel.

  5. Click Create pipeline schedule and verify that the schedule and the pipeline branch is set properly in the next screen.

    The image shows a dark-themed GitLab interface confirming the successful scheduling of a pipeline, with a notification at the top. Below, the table lists the scheduled pipelines with columns for Description, Target, Last Pipeline, Next Run, and Owner. The pipeline named Coa Refresh Test is scheduled to run on the main branch, with the next run in 6 hours, and options to run the pipeline immediately, edit, or delete it.