Skip to main content

Refreshing Your Pipeline Using the Coalesce Scheduler

The Coalesce Scheduler gives you the ability to schedule data pipeline refreshes directly in the Coalesce app. You no longer need to use a third-party solution simplifying your IT dependency and allowing you to generate data insights faster.

Before You Begin​

  • You must have an Environment Admin role.
  • The Environment must have credentials added. The Scheduler uses the credentials of the user that created or last modified the Scheduled Job.
  • A Job must be deployed before it can be scheduled. The Coalesce Scheduler only runs deployed Jobs, not workspace-only changes that are not yet committed and deployed. Learn more about deploying your pipeline.
Deleting or Changing Credentials

If you change credentials, including changing authentication type, you'll need to edit each Job to use the new method. If not, the Jobs will fail.

Subgraph Changes, Commits, and Scheduled Runs​

When you adjust a subgraph and that change should affect what a Job refreshes, commit the Job (together with your other Project changes) to version control, then deploy to the environment. Scheduled runs use the deployed Job definition. Until you commit and deploy, the Coalesce Scheduler keeps running the last deployed version.

For how subgraphs relate to Jobs and selector queries, see Using Subgraphs to Build Your Pipeline. For the commit and deploy sequence, see Creating and Run Jobs.

How to Schedule a Job​

  1. In your Coalesce App, go to the Deploy page.

  2. Next to any Job, click the dropdown and select Create Job Schedule. You can also click the Deploy button next to the environment.

    A dropdown menu under a job called Test Refresh Job within a development environment. The menu offers options to run the job, create a job schedule, or view existing schedules.

  3. Configuration: Fill out the Job Schedule form. If you click on the Job to create a Job schedule, the Project, Environment, and Job Name will be pre-filled.

    1. Project - You’ll be able to create Jobs for any Project you have Environment Admin access to.
    2. Environment - Select the Environment in the Project.
    3. Job Name - Select an existing Job in the Environment.
    4. Job Schedule Name - Choose a unique name to make it easy to identify later.
    5. Cron Expression - Enter a Cron expression in UTC time for the Job frequency.
    6. Automatic run on save - When this option is on, the Job runs once as soon as you save the schedule, then continues on the cron schedule. Turn it off if the first run should wait until the next scheduled time instead.
    7. Retry on Failure - Toggle to retry the entire Job if it fails. This will retry the entire Job.
      1. Number of Retries - Enter the number of times a Job will attempt to run if it fails the first time.
      2. Time Between Retries - Enter the number of minutes between each retry.
      3. Retry Job From - Beginning will run the entire Job again. Point of Failure will retry the Job from when it failed.
    This image shows a web interface for editing a job schedule in the Coalesce platform. The interface includes various configuration options such as project, environment, job name, schedule frequency, and retry settings for managing automated jobs.

  4. Notifications: Coalesce will send an email notification for successful, retries, and failed Jobs.

    1. Email on: Choose if you want emails to be sent for Success only, Errors only (includes retries), or both.
    2. Email Addresses: Each email address should be entered on a new line.
    Editing a job schedule in a development environment. It includes fields for the project, environment, job name, and job schedule name, along with a cron expression for scheduling

  5. Save the Job. You’ll be able to see the Job on the Job Schedules page.

Detailed view of a specific job schedule within a development environment, including the last execution time, next execution time, and the run status marked as Successful.
Refresh the Page

Refresh the page to see the status of a Scheduled Job.

View Scheduled Jobs​

You can view Scheduled Jobs in two ways.

  1. Select View Schedules next to the Job you want to view the schedules for.

    A dropdown menu under a job called Test Refresh Job within a development environment. The menu offers options to run the job, create a job schedule, or view existing schedules.

  2. Select View Schedules Jobs next to Deploy to see all Scheduled Jobs for the Environment.

A dropdown menu next to the deploy button offers shows options for environment settings, viewing scheduled jobs, and documentation

You’ll be able to view a list of Scheduled Jobs on the Jobs Schedule page.

Job schedules within a development environment, showing details like project name, environment, job name, schedule name, last execution, next execution, and run status. The list shows three different job schedules with their respective statuses and times

Edit Scheduled Jobs​

  1. On the Jobs Schedule page, in the Actions column, select Edit.
  2. You can edit the Job Schedule Name, Cron Expression, and Automatic run on save. If you need to edit the Project or other information create a new Job Schedule.
Editing a job schedule in a development environment. It includes fields for the project, environment, job name, and job schedule name, along with a cron expression for scheduling
Editing Jobs Created By Another User

Editing Scheduled Jobs created by another user, will use your Coalesce and Data Platform credentials to run the job

Scheduled Jobs Notifications and Alerts​

Notifications can be sent using email. You can configure emails to be sent for Success, Error, or Both.

You can also edit the notifications at any time by editing the Job Schedule.

Editing job schedule notifications, allowing the user to configure email notifications for job successes, errors, or both. It includes fields to input multiple email addresses.

Notifications are sent for:

  • Each successful Job Schedule.
  • Each failed Job Schedule.

Pause or Resume Scheduled Jobs​

If you need to stop a schedule without removing it, pause it instead of deleting the Job Schedule.

  1. On the Jobs Schedule page, in the Actions column, select Pause.
  2. To start runs again, select Resume from the same menu.

While a schedule is paused, Next Execution shows Paused. The Job does not run until you resume.

Delete Job Schedules​

To delete a Job Schedule, go to the Jobs Schedule page, in the Actions column, select Delete. Pausing is a better option when you plan to use the same schedule again soon.

  • You can delete a Scheduled Job at any time.
Deleting a Job

If you delete a Job attached to a schedule, then commit the deleted Job change to the deploy branch, then try to deploy, you’ll get a warning message in the Review Plan step. The Jobs Schedules for the deleted Job will be removed at the next scheduled run and won’t be part of the deploy.

A deployment review screen with a warning message stating that certain job schedules will be deleted and will not run after deployment. A specific job schedule named Test delete is highlighted as being marked for deletion

Scheduled Job Status and Results​

On the Job Schedule page, you can see:

  • Last Execution - Last time the Scheduled Job was run displayed in system time.
  • Next Execution - Next Scheduled Job run time, or Paused when the schedule is paused.
  • Run Status - Most recent status of the Scheduled Job.
    • Successful: The Job ran without issue.
    • Running: Job is currently running.
    • Failed: The Job failed to run.
    • Test failed: The Job ran, but the Node tests failed.

To view the results click on the Run Status of any scheduled job.

Results of a specific job schedule within the development environment, including details such as the project, environment, job name, schedule name, last execution, next execution, and run status, with a successful execution indicated. The successful text is higlighted

Scheduler Permissions​

Important Scheduling Information​

  • You can create multiple schedules for each Job.
  • Refreshed All Nodes and Ad-Hoc jobs aren’t part of the Job Scheduler and can only be run using the CLI or API.
  • Schedules use Cron to run and they are in UTC time.
  • By default, a new or edited schedule runs once when you save if Automatic run on save is on. Turn that option off so the first run waits for the next cron execution.
  • Selecting Run Job from the Jobs menu will run the Job one-time. It will not change the Scheduled Job.
  • You can pause a Scheduled Job from the Actions column on the Jobs Schedule page instead of deleting it or changing the cron expression.
  • Scheduled Jobs run in parallel. If more than one Job is scheduled for the same time, they will run at the same time. If there is an ad hoc refresh during a Scheduled Job, they will run at the same time.
  • Scheduled Jobs aren’t committed to your Git provider.
  • The Coalesce Scheduler only runs deployed Jobs. Subgraph or Job changes in the Workspace do not affect scheduled runs until you commit, deploy, and the environment has the updated definition.

Resources​