Coalesce Best Practices

Before Starting With Coalesce

• You'll need access to Snowflake.
  • The data in Snowflake needs to be set to READ for sources. A source is the data Coalesce will read for data transformations.
  • Targets in Snowflake need to be READ/WRITE. A target is where Coalesce will write data transformations.
• Google Chrome is the only supported browser.
• Sign up for Coalesce.
• To follow tutorials, install the Snowflake sample data. It's included if you're on a Snowflake trial.

Organization Setup

• Networking
  • Inbound traffic from Coalesce must be allowed.
  • Outbound traffic to Coalesce must be allowed.
• Review the requirements for virtual desktops (optional).
• Add users
  • An Organization Administrator will need to add users.
  • Review the Authentication Methods for Authenticating to Coalesce and choose the best one for your organization.

Version Control

• Configure Version Control
  • Choose a provider.
  • Each user will need their own provider account and create a personal access token for Coalesce.
  • Each user will need to be in the organization's account.
  • It's recommended to create a new repository for Coalesce.

Git Commits

• Make frequent commits using meaningful commit descriptions.
• Keep your commits simple. They should be for solving a single unit of work or single task. This makes it easier to revert changes.
• Ensure that all members of the development team are aware that a commit is planned, so that all code for the Workspace is in a ‘commit ready' state.
• Consider allocating a single developer the task of performing commits to the Main branch, to avoid any possible conflicts.
• Avoid making breaking commits to your main Git branch.

📘

Having Trouble Making Commits

If you cannot commit your changes in a Workspace to Git, contact Coalesce support for help. Potentially overwriting your live metadata with a previous commit will cause you to lose all recent uncommitted development.

Git Branches

• Use a branching strategy.
• Use branches for development to keep work in progress away from the main branch.
• Only deploy to target Environments from a ‘Main' branch, rather than from feature branches.
• One person at a time should work on a branch.

Workspaces and Git

• Do not have more than one instance of the Git modal open for a single Workspace, at the same time.
• Never develop directly in your main development Workspace.
• Never merge into a Workspace which has uncommitted changes.
• Never change the Git repo settings when having uncommitted changes in ANY Workspace.
• All development Workspaces should be ideally mapped to different schemas (except perhaps source nodes). You can't have a Workspace AND Environment on the same set of Snowflake schemas, unless the Workspace is used as read-only (which cannot be enforced).
• All Environments should be mapped to different schemas, except for source nodes in some cases.

Project Setup

• An Organization Administrator will need to create a new Project and configure Git during the process.
  • One repository per project is recommended.
• Create a newWorkspace in the Project. This will be the main branch.
• Create your Environments.
  • Each environment should have its own Database and Schema. As a feature flows from a main development Workspace to Test, QA, and finally to the Prod environment, the storage mappings should point to the Snowflake locations where you want to create the objects in question in your pipeline.
  • A good idea is to create environments for DEV, QA, and Production matching the Storage Mappings.
  • You'll have to connect each environment to Snowflake. .
• Have each user join the Project they'll be working on and connect to their Git accounts.

Workspaces

• Create a Workspace strategy. Some approached you can take are:
  • Use one Workspace per branch. Each time you create a new branch, create a new Workspace and delete old ones.
  • Use one Workspace per user. Each user works in their own Workspace. They manage their branches from the Workspace.
  • Create a separate Workspace for each feature branch. This ensures that developers work within distinct Snowflake locations, preventing conflicts and streamlining the development process.
• Decide how you want to connect your Workspace to Snowflake.
• Configure your Storage Location and Mappings. A good rule is to create target schemas in Snowflake for DEV, QA, and Production. Then map them in Coalesce.
• When feature dev in a specific Workspace is complete, it should be merged into the main branch and checked out by the Main Workspace when preparing for deployment into higher Environments.
• It is best practice to deploy to higher (Production, UAT, Testing) Environments from the main Git branch in the Main Workspace.

Deploy Your Pipeline

• Make sure your environment has the following configured:
  • Storage Location and Mappings
  • Snowflake
  • Git
• The work has been committed to the branch you want to deploy or refresh.
• The target databases and schema are configured in Snowflake.
• Configure your Parameters.
• Decide if you want to deploy using the CLI or Coalesce App.
• Review ways to Schedule Your Pipelines.

Refresh Your Pipeline

• Make sure your environment has the following configured:
  • Storage Location and Mappings
  • Snowflake
  • Git
• Create your Jobs.
• You must deploy with your created Jobs before you can refresh.
• Configure your Parameters.
• Decide if you want to refresh using the API or CLI.
• Review ways to Schedule Your Pipelines.