Skip to main content

Pipeline Documentation

Documentation is an essential component of building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. Coalesce automatically produces and updates documentation as developers work.

Accessing Generated Documentation​

Generated documentation for a Workspace or Environment can be accessed in three different ways.

Workspace and Environments​

Both Workspaces and Environments - From the top of the user interface, through the Docs link.

Pipeline documentation from the top of the user interface, through the Docs link

Workspaces​

Only Workspaces - From the Projects Dashboard, through the Docs icon.

Pipeline documentation from workspaces

Environments​

Only Environments - From the Deploy Dashboard click on the ellipses and then View Docs.

Pipeline documentation in deploy dashboard

Interface Overview​

The new documentation interface consists of three sections.

Generated docs interface

Users can search for Node names, physical types, databases, schemas, and storage locations. Node names can be searched for using any portion of the Node's name, while all other objects are searched using a prefix search only. For example, if the storage location is PROD_RAW, searching for RAW will not yield any results. On the other hand, if a Node is named STG_CUSTOMER, searching for either STG or CUST will be effective.

2. Filters​

The Filters dropdowns allow the user to limit the Nodes currently visible in the Results List. For example, if one wanted to see only Nodes of type Stage, then they would select Node type > Stage, and the Results List will be updated in real time.

3. Results List​

This section will have the results, which come from a combination of search terms and filters used.

Sharing Results​

Search terms and filters automatically adjust the URL, which makes results easy to share, as other users can use the same URL to access the same results.

Exporting Documentation​

The Coalesce documentation can't be exported. The Command Line Interface and API can be used to programmatically gather information about your Coalesce organization, such as a list of your Environments with their details, Node contents, etc.

Documentation Walkthrough​

You’ll learn more about Coalesces' documentation feature and how to navigate each page.

Nodes Landing Page​

The landing page helps you filter and focus on the specific Nodes you wish to view. In this example, the filter for Dynamic Table Work was added and returns three Nodes of that type. Clicking on a Node name will take you to the documentation for that Node.

Dynamic Tables Example workspace with filters, table names, schemas, and development environment tags displayed in a list.

Node Information​

This section provides navigation and column documentation for the Node. Expand each section by clicking the arrows, or use the breadcrumbs to navigate.

Dynamic Tables Example page showing DT_WRK_DT_CUSTOMER_NATION_REGION_DT_ORDERS_LINEITEM with metadata including Work and DEV tags.

The header shows:

  • Node name - DT_WRK_DT_CUSTOMER_NATION_REGION_DT_ORDERS_LINEITEM
  • Node Type - Dynamic Table Work.
  • Database - TATIANA_DOCS_DYNAMIC_TABLES
  • Table - DEV
  • Storage Location - WORK

Information is organized into the following sections.

Overview​

General information about a Node.

Node Overview showing logical type as Dynamic Table Work, physical type as Dynamic table, database _DOCS_DYNAMIC_TAB_WORK, schema DEV.

Columns​

Shows all the columns in a Node.

Columns table with two entries: C_NAME (VARCHAR 25) and L_QUANTITY (NUMBER 12,2), each with a View option and pagination controls.

Column View​

Click View next to a column to display its information, including the column name and its parent.

Data flow diagram with four connected tables: CUSTOMER, STG_CUSTOMER_REGION_NATION, DT_CUSTOMER_NATION_REGION, and DT_WRK_DT_CUSTOMER_NATION_REGION.

The column lineage can be navigated by clicking on the ellipses and clicking Go to column documentation. It will then take you to the documentation for that column in the preceding Node. You can continue to do that for each column in the lineage. Return to the Node Overview by clicking the Node name.

Lineage​

This shows the Node Lineage. Use the controls to move around the Node.

Node lineage diagram showing data flow from CUSTOMER, REGION, NATION, LINEITEM, and ORDERS to staging and transformation tables.

Create Scripts​

Any scripts that ran as part of the Create function. If there is more than one, it will be another tab.

SQL script for creating or replacing a stage table named STG_SUPPLIER with defined columns and a comment about supplier data.

Run Scripts​

Any scripts that ran as part of the Run function. If there is more than one, it will be another tab.

Run Scripts section with SQL code to truncate the STG_SUPPLIER table if it exists in the _DOCS_TESTING.DEV schema.

Configuration​

Any configuration options available for the Node. This can vary depending on the Node.

Configuration options table with settings: Truncate Before, Enable Tests, Distinct, Group By All, Order By set to False; Pre-SQL and Post-SQL as N/A.