This page will help you quickly get started with Coalesce, connecting your database and setting up a staging node with a simple data transformation. Optionally, we will also walk you through the creation of Dimension and Fact tables.
Before You Start
- If you haven't yet signed up for a Coalesce account, you can do so here.
- Make sure you're using Google's Chrome browser, as other browsers are not officially supported.
- Have your Snowflake login information handy. Don’t have a Snowflake account yet or not ready to connect your company’s Snowflake to Coalesce? Sign up for a free trial here and use Snowflake’s provided sample data.
When you first sign in, you'll be presented with the Workspaces Dashboard, where you should choose and launch a workspace.
Once you've done this you'll be presented with the Build interface. This is where you'll spend most of your time creating nodes, building a graph with them, and transforming your data. The other interface is the Deploy interface, which will be used to push your pipeline to other environments (QA, Production, etc.), including your graph and other metadata.
The Problem Scanner
Note that on a brand new account, the Problem Scanner will show a few action items. You can disregard those as most will be resolved by the end of this guide.
- Click on Build Settings, which is represented by a cogwheel icon toward the bottom left
Go to Development Workspaces and edit your current Workspace by clicking the pencil icon to the right of it.
From the Edit Workspace screen → Settings → Account → Add your Snowflake URL
- From the Edit Workspace screen → User Credentials and fill out the form with your Snowflake login
- Click Test Connection to ensure your credentials work as expected
- Click Save
You've now connected Coalesce to your Snowflake instance!
A storage location is a logical name you provide to represent a database and schema (in Snowflake) and you will need them to make use of the Workspace you configured earlier.
- Go to Build Settings → Storage Locations and click on New Storage Location to create 2 new locations - a source and a target.
- You can name them anything you'd like, in this guide we are calling one SRC and the other TARGET.
- Make sure to set the TARGET as the default Storage Location.
- Edit your Workspace → Storage Mappings
- From here enter the actual databases and schemas that will correspond to each Storage Location. For this guide we will be using Snowflake's sample schema TPCH_SF1 as our data source. Feel free to use any schema you'd like as the target for the transformed data.
Now it's time to add Sources to the graph. The graph node view is where you'll configure Nodes that will transform your data. Below is an example of a graph with several nodes -
To add Source Nodes take the following steps:
- Expand Nodes from the Left Sidebar (if not already open)
- Click on the + sign → Add Sources
- Choose your source tables on the left and click Add Sources on the bottom right to add the sources to your pipeline
- You'll now see your graph populated with some Source Nodes
Now that you have a few Source Nodes on your graph, it's time to add a Stage Node. Stage Nodes are intermediate nodes in the graph where you prepare the data by applying business logic.
- Add one or more Stage Nodes by right clicking your
NationSource Node → Add Node → Stage Node. Note that you can select multiple Source Nodes by Shift+clicking them and then add multiple Stage Nodes simultaneously.
- Double click on the Stage Node or right-click → Edit to open up the Node Editor
- Open Node Properties on the right and change the Location to the Target you configured earlier
- Click Create to create a table in Snowflake
- Click Run to populate the table. Note that you haven't transformed the data yet!
- Edit the Transform field in the Mapping grid by double clicking in the transform field of the
N_NAMEcolumn. Try a simple transform like
LOWER()and the name of your column, or you can use the syntax
- Click Run again to transform the data
Any Snowflake SQL transform can be used to transform your data.
- You'll see a preview of your transformed data in the lower half of the screen, but feel free to take a peek in your Snowflake database to confirm.
Congratulations! You've connected your database and applied a basic transformation to your data. Feel free to continue experimenting with some of the other node types below.
Now let's create a simple slowly changing dimension (Type 2) — an industry standard for tracking historical data by creating multiple records for a given natural key.
- Create a new Stage Node from the CUSTOMER Source Node
- Go into this new node to Create and Run
- Return to the main graph and create a Dimension node from the
STG_CUSTOMERnode. By default, Coalesce creates a Type 1 Dimension. In this guide we will be making a Type 2 Dimension to track historical data of a column.
- Go into the new
- Open up Options on the right side
C_CUSTKEYas a business key by selecting it and clicking the arrow to move it to the right
- Scroll down the options pane to Change Tracking and select columns
C_PHONE, clicking the arrow to move them to the right
- Now Create and Run the
Type 1 vs Type 2
In the Dimension Node, if no Change Tracking columns are selected, the node will act as a Type 1 Dimension. If Change Tracking columns are selected, it will act as a Type 2.
You have now finished creating a Type 2 Dimension table. Next we will make a Fact Table.
Now let's create a fact table.
- Create a new Stage Node from the
- Open the new
STG_ORDERSnode and delete all the columns except for
- Select the
DIM_CUSTOMERnode on the left side, then select
DIM_CUSTOMER_KEYand drag it into your
- Go to Join in
- Delete the existing text
- Click Generate Join and then Copy to Editor
- Replace the
- Create and Run the
- Create a Fact Node from
- Open the new
- Open Options → Business Key → add
- Create and Run the
You have now made a fact table! You can run this query in Snowflake (adjusting your schema and databases from
MY_SCHEMA to the ones in your environment) to confirm.
select DIM.C_NAME CUSTOMER_NAME, sum(FCT.O_TOTALPRICE) TOTAL_PRICE from "MY_DB"."MY_SCHEMA"."FCT_ORDERS" FCT inner join "MY_DB"."MY_SCHEMA"."DIM_CUSTOMER" DIM on FCT.DIM_CUSTOMER_KEY = DIM.DIM_CUSTOMER_KEY group by DIM.C_NAME;
Updated 4 days ago
Now that you've finished the Quick Start Guide, you can try using other node types, backup your work using our git integration, or try deploying your pipeline to an environment.