Step 3: Create a Workspace

Before creating a Workspace, you should make sure you have a Project created with version control. You can choose from username and password or OAuth.

A Workspace is a sandbox environment where you can complete the development of your data, refine, and perform preliminary validation before merging them into the codebase. When creating a Workspace, you'll add your Storage Locations, Git URL, and connect to your data platform.

Workspace Best Practices

• Create a Workspace strategy. Some approached you can take are:
  • Use one Workspace per branch. Each time you create a new branch, create a new Workspace and delete old ones.
  • Use one Workspace per user. Each user works in their own Workspace. They manage their branches from the Workspace.
  • Create a separate Workspace for each feature branch. This ensures that developers work within distinct locations, preventing conflicts and streamlining the development process.
• Decide how you want to connect your Workspace to a Data Platform. Review below.
• When feature development in a specific Workspace is complete, it should be merged into the main branch and checked out by the Main Workspace when preparing for deployment into higher Environments.
• It is best practice to deploy to higher (Production, UAT, Testing) Environments from the main Git branch in the Main Workspace.

info

Before you begin, make sure you have administrator access to your chosen platform and the necessary permissions to create authentication credentials.

Just need a quick reference for authentication? Look at the Connection guides for your data platform.

Choose Your Platform

Select the platform you want to connect to:

BigQuery (Private Preview)

An admin or a user with the ability to assign permissions is needed for BigQuery.

Set Up Project Level Permissions.

1. Make sure you're in the project you want to add the service account to.

2. Go to IAM & Admin > Service Accounts in Google Cloud.

   

3. Click Create service account.

   

4. Fill out the required Service account ID and any other information required by your company's policy. Click Create and Continue.

5. Search for the role BigQuery User. Click Continue.

   

6. You can skip Principals with access. Click Done. Move on to setting data set level permissions.

Set Up Dataset Level Permissions.

1. Go to BigQuery Studio.

2. In the Explorer, find the data set you want to add to Coalesce.

   

3. In the data set, click Share > Manage Permissions.

   

4. Click Add principal.

5. Search for your service account name. It should look similar to name@projectname.iam.gserviceaccount.com.

6. Then assign roles as BigQuery Data Editor.

   

7. Go to IAM & Admin > Service Accounts in Google Cloud.

   

8. Click on the service account created for Coalesce.

9. Click Keys.

10. Then Add key > Create new key.

    

11. Download the JSON file.

Create Your Workspace.

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

5. Click Next.

   

6. Add your BigQuery Credentials

7. Upload the JSON file from Google Cloud IAM. Then click Test Connection.

   

8. Click Next.

9. Set Up Version Control

10. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

11. Then give the new branch a name.

12. Click Next.

    

13. Storage Location and Storage Mappings

14. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

15. Click Create Workspace.

Databricks

Before starting with Databricks authentication, make sure the following is in place:

• Your workspace must be using the Unity catalog.
• Must be a Databricks account admin create to OAuth applications.
• Databricks users must be given the permission to create a Personal Access Token (PAT) for Token authentication.

Token Authentication

OAuth

We recommend using OAuth for authentication.

Tokens or Personal Access Tokens (PAT) are used in Databricks to authenticate at the workspace level.

In this section, you get your access token from Databricks and then add it to Coalesce as part of creating a Workspace.

Get Your Databricks Token.

1. Log into your Databricks account.
2. Go to your username at the top and select Settings.
3. Under User, select Developer.
4. Next to Access Tokens, click Manage.
5. Click Generate new token.
6. Write a description for the token and choose the lifetime.
7. Make sure to save the access token somewhere secure. It can’t be recovered or shown again.

Databricks Access Token Page

Databricks PAT

Read the Databricks documentation on Databricks personal access token authentication to learn more about managing your access tokens in Databricks.

Create Your Workspace.

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

5. Click Next.

   

6. Connect Databricks to this Workspace

7. Enter your Databricks Account URL.

8. Click Next.

   

9. Add your Databricks Credentials,

10. Enter the token and then click Retrieve Warehouses. Select the one you want to use.

11. Click Next.

    

12. Set Up Version Control

13. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

14. Then give the new branch a name.

15. Click Next.

    

16. Storage Location and Storage Mappings

17. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click Create Workspace.

OAuth Machine-to-Machine (M2M)

Machine-to-Machine is used for automation. You'll need to create a Service Principal.

1. On the Databricks Account Console, go to User Management and select Service Principals.

   

   Databricks User management

2. After giving the service principal a name, you'll grant the right permissions, then create the client ID and client secret.

   1. The service principal should have Workspace, Warehouse, and Catalog permissions.

3. Click on the service principal and go to the Permissions tab and click Grant Access. Add the service principal you just created as a User.

   

   Add the new service principal as user

4. Go to the Databricks Catalog and select the catalog the service principal should have access to.

5. Click Grant.

   

   Click Grant on the catalog

6. Choose the service principal and then select Data Editor.

   

   Grant the Data Editor Role

7. Then go to SQL Warehouses, and select the warehouse the service principal should have access to.

   

   SQL Warehouse Permissions

8. Click Permissions, and select the service principal from the list. Set it to Can Use.

   

   Permission Level is Can Use

   Service Principal Missing From List

   If your service principal is missing from the list, go to your user Settings > Identity and access > Service principals, and add it there too. Then go back to the SQL Warehouse step.

9. Go to the Account console in Databricks.

10. Select User management > Service principals.

11. Click the one you created, then click Generate secret.

    

    Generate Your Client ID and Client Secret

12. Choose how long the token should be valid for, then click Generate.

13. Save the Client ID and Client Secret for configuring in Coalesce.

Create Your Workspace.

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

5. Click Next.

   

6. Connect Databricks to this Workspace

7. Enter your Databricks Account URL.

8. Click Next.

   

9. Add your Databricks Credentials.

10. Change the Authentication Type to Machine-to-Machine.

11. Enter the Client ID and Client Secret, then click Retrieve Warehouses. Select the one you want to use.

12. Click Next.

    

13. Set Up Version Control

14. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

15. Then give the new branch a name.

16. Click Next.

    

17. Storage Location and Storage Mappings

18. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

19. Click Create Workspace.

OAuth User-to-Machine (U2M)

User-to-Machine is used for your daily work access. If you want to set up any automation, use Machine-to-Machine. You will need to create a App Connection and get the client ID and client secret along with the URL and path.

1. Go to the Account console. If you don't have access, you'll need to reach out to your Databricks administrator.

2. Then go to Settings > App Connections and click Add connection.

   

   Databricks - Settings > App connections

3. Give the app a Name:

   1. Redirect URL is your Coalesce account URL at oauthredirect. This is usually the URL you use to log into Coalesce. For example, https://<COALESCE_APP_DOMAIN>/oauthredirect, https://app.coalesce.io/oauthredirect.

      

      Databricks - Add Connection

   2. Change the Access Scopes to All APIs.

   3. Generate client secret is checked.

   4. You can leave TTL as the default or change it.

4. Click Add.

5. A Client ID and Client Secret will pop-up. Save them for later.

Create Your Workspace.

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

5. Click Next.

   

6. Connect Databricks to this Workspace

7. Enter your Databricks Account URL.

8. Toggle Databricks OAuth (User-to-Machine) on. This will open a window where you can enter the Client ID and Secret.

9. Coalesce will verify the connect and then you can click Next to move to the next step.

   

10. Add your Databricks Credentials

11. Select the SQL Warehouse you want to use, then click Next.

    

12. Set Up Version Control

13. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

14. Then give the new branch a name.

15. Click Next.

    

16. Storage Location and Storage Mappings

17. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click Create Workspace.

Snowflake

Before you begin, you'll need your Snowflake account URL.

1. Follow the instructions in Snowflake for Finding the organization and account name for an account.
2. On the Account Details screen, copy the Account/Server URL. You'll need the first part. For example, if your URL is ABC123-1234567.snowflakecomputing.com, you'll need ABC123-1234567.

---

OAuth

Permissions Requirement

Only a Snowflake ACCOUNTADMIN role or a role with the global CREATE INTEGRATION privilege can create security integrations.

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

   

5. Connect Snowflake to this Workspace

6. Enter your Snowflake Account URL.

7. Then enable Snowflake OAuth.

8. A popup will include the code needed to create the Client ID and Client Secret. First run Create Security Integration, then run Fetch ClientID and ClientSecret.

   

   The OAUTH_REDIRECT_URI will match your Coalesce app domain.

9. The next screen will ask you to enter your Role (SYSADMIN) and Warehouse if they are required. Click Authenticate.

10. You'll be asked to sign in to your Snowflake instance.

11. Coalesce will confirm the connection, then click Next.

    

12. Set Up Version Control

13. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

14. Then give the new branch a name.

15. Click Next.

    

16. Storage Location and Storage Mappings

17. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click Create Workspace.

Key Pair Authentication

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

   

5. Connect Snowflake to this Workspace

6. Enter your Snowflake Account URL.

7. Click Next.

   

8. Add your Snowflake Credentials

9. Choose Key Pair as the Authentication Type.

   1. Go through Snowflake’s key pair authentication steps to generate your keys.
   2. Make sure to generate a private key and save it for use in Coalesce.
   3. Generate your public key using the private key created for Coalesce.
   4. Assign the public key to your Snowflake user.

10. Add your Private Key and Private Key Passphrase.

11. Click Test Connection.

    Adding Your Private Key

    When entering your private key, make sure it's formatted properly. It must include the full private key including the lines BEGIN ENCRYPTED PRIVATE KEY and END ENCRYPTED PRIVATE KEY.

    -----BEGIN ENCRYPTED PRIVATE KEY-----
    ...
    -----END ENCRYPTED PRIVATE KEY-----

12. After authentication is successful, click Next.

    

13. Set Up Version Control

14. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

15. Then give the new branch a name.

16. Click Next.

    

17. Storage Location and Storage Mappings

18. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

19. Click Create Workspace.

Username and Password (Not recommended)

1. Select the Project you want to create the Workspace in.

2. Click Create Workspace.

3. Workspace Details

4. Give your Workspace a Name and Description(optional).

   

5. Connect Snowflake to this Workspace

6. Enter your Snowflake Account URL.

7. Click Next.

   

8. Add your Snowflake Credentials

9. Enter your Snowflake username and password.

10. Enter the Role and Warehouse.

11. After authentication is successful, click Next.

    

12. Set Up Version Control

13. Workspaces let you work on a branch. This is the version control repo you created in Setup Version Control and added to your Project. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.

14. Then give the new branch a name.

15. Click Next.

    

16. Storage Location and Storage Mappings

17. Add Storage Locations and Storage Mappings to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click Create Workspace.

---

What's Next?

Review the Coalesce Fundamentals to learn more about:

• Projects
• Workspaces
• Add Your Data

---