Skip to main content

Environments

Environments are used for deploying your data pipeline to your data platform. They manage efficient deployment strategies and allow you to configure both your Workspace and Environment. This lets you complete your development in a Workspace and then deploy to any Environment you want, such as production or QA, with each having different Storage Locations, Storage Mappings, Parameters, and other options.

A Coalesce architecture diagram with two panels. The left panel shows an organizational structure with an administration layer and various department projects, while the right panel details the Central IT Compliance workflow, displaying feature workspaces that feed through git and deployment pipelines into production, QA, and dev environments.

Key Concepts​

  • Workspace: Where you develop and test your data pipeline.
  • Environment: Where you deploy or refresh your production data pipeline.
  • Development vs Deployment: No development happens directly in an Environment - all changes come through defined Git branches and commits.

Why Environments Are Important​

Environments serve several purposes in the data pipeline lifecycle:

  • Separation of Development and Production: Environments let you to keep development work isolated from production systems, preventing accidental impacts on live data.
  • Deployment Control: They enable controlled deployment of changes through a structured process, ensuring that only tested and approved changes reach production.
  • Environment-Specific Configuration: Each environment can have different configurations (credentials, parameters, storage locations) appropriate for its purpose.
  • Organized Testing Progression: Environments are a structured was of testing progression from development to QA to production.

Common Environment Types​

  • Development: For initial testing after development in Workspaces,
  • QA/Test: For quality assurance testing before production.
  • Staging: For final verification in a production-like setting.
  • Production: The live environment where transformations run on actual business data.

Important Notes​

  • Objects are created in a steady state through Git branches and commits.
  • Data loading occurs on demand, with schedules/triggers, or an automated process such as CI/CD pipeline.
  • Column changes use ALTER instead of DROP/CREATE.
  • Deleted objects are automatically cleaned up (DROPPED) in your data platform.
  • No development happens in Environments, work is done in Workspaces. Each Workspace can have multiple Environments.

Environment Settings Overview​

Environment Settings

To edit or create an Environment, go to Build Settings > Environments.

  1. Settings - Includes general account information such as the Environment name and the connection URL. Take note of the Environment ID. You'll use this when making API requests to see information about the deployment. You can also delete your Environment here.
  2. User Credentials - Configure how you want to connect to your Data Platform. If you want to use OAuth (recommended), you'll also need to add your OAuth Settings.
  3. Storage Mappings - To deploy or refresh your environment, you need to have your Storage Mappings setup. Storage Mappings are represent the physical locations in your Data Platform. This is where your data pipeline will deploy. Learn more about Storage Mappings.
  4. Parameters - Parameters are defined as a JSON blob set by the user that can be accessed in the metadata during template rendering. Learn more about Parameters.
  5. OAuth Settings - If you're using OAuth to connect to your data platform, you'll need to configure OAuth on your data platform.

What's Next?​

  • Deployment Overview - Learn how to deploy your data transformations to the environments you've configured.
  • Refresh Your Pipeline- Understand how to execute transformations and update data in your deployed environments.
  • Parameters - Discover how to make your pipeline flexible across different environments using parameters.
  • Step 9: Create Your Environments - Go through the steps to creating a new environment.