Storage Locations and Storage Mappings

Storage Locations and Storage Mappings are closely related but serve different purposes in managing and structuring data within Coalesce.

Storage Location

A Storage Location in Coalesce represents a logical destination for the database objects you use as sources and create within the platform. It acts as a container or grouping for these objects, such as views and tables, allowing you to organize and manage them effectively. Each Coalesce node, which represents a specific database object, is mapped to a single Storage Location. However, a single Storage Location can be mapped to multiple Nodes, providing flexibility in organizing your data.

📘

Storage Location Mappings

Storage location mapping definitions are committed in Git for Environments, not for Workspaces.

Create a Storage Location

1. Launch your workspace.
2. Go to the Build Settings.
3. Click on Storage Locations.
4. You can create as many Storage Locations as you need. It's a good idea to have a source data storage location and a transformation storage location. In this example, you'll create Source and Transformation. You can create more Storage Locations if needed. A best practice is to separate your nodes into different storage locations that represent different groupings of datasets and to name them appropriately.

Edit a Storage Location

To set as default or delete the storage location, click on the edit icon on the storage location card. You can configure default storage locations for specific node types in specific workspaces. See Node Types for more details.

For a quick overview and setup instructions for your first Storage Location, see the following video.

Storage Mapping

Storage Mapping is a physical destination within your database, specifically in Snowflake, that is defined for a Storage Location in a given Workspace or Environment. It represents the actual database and schema where the Storage Location resides. It allows you to define the exact location where your database objects will be stored within your Snowflake account.

By using Storage Locations and Storage Mappings in Coalesce, you can achieve granular control over the organization and storage of your database objects. You can map different Storage Locations to different physical locations, enabling you to tailor the storage of your data based on specific requirements.

For example, in a development workspace, you can map each of your Storage Locations to the same physical location, effectively placing all database objects in a single schema used as a scratch working area.

Meanwhile in production, you can map each Storage Location to a different physical location for each storage location, allowing for the use of Snowflake roles on specific schemas to control access to objects.

Create a Storage Mapping

When adding sources in the build interface, only physical locations that are mapped to a storage location for that environment/workspace are available to be added. This means that you will need to define a storage location for source tables and map that to a physical location for that environment/workspace. Make sure you have configured your Storage Locations and connected to Snowflake. Then you can map the physical locations in Snowflake to your Storage Locations you created in Coalesce.

1. Go to Build Settings > Development Workspace in your Workspace.
2. Edit your Workspace > Storage Mappings.
3. Because you created two Storage Locations previously, there should be two Storage Mappings available. You can create as may mappings and locations as needed.
4. Now you can map your database to the storage location. Choose the data and schema you want to map to Source and the data and schema you want to map to Target.

For setup instructions and a quick overview on Storage Mappings, check out the following video.

Override Mapping Values

To input database and schema objects manually, instead of choosing them from the dropdown menu, activate the Override Mapping Values option. This step may be necessary when mapping to a database or schema for which you lack access permissions.