Skip to main content

Documentation index: llms.txt. This page is also available as markdown: append .md to this URL or send Accept: text/markdown.

Refreshing Your Pipeline Using the Coalesce Scheduler

The Coalesce Scheduler gives you the ability to schedule data pipeline refreshes directly in the Coalesce app. You no longer need to use a third-party solution simplifying your IT dependency and allowing you to generate data insights faster.

Before You Begin​

  • You must be an Environment Admin, a Project Admin for the Project, or an Org Admin.
  • The Environment must have credentials added. The Scheduler uses the credentials of the user that created or last modified the Scheduled Job.
  • A Job must be deployed before it can be scheduled. The Coalesce Scheduler only runs deployed Jobs, not workspace-only changes that are not yet committed and deployed. Learn more about deploying your pipeline.
Deleting or Changing Credentials

If you change credentials, including changing authentication type, you'll need to edit each Job to use the new method. If not, the Jobs will fail.

Subgraph Changes, Commits, and Scheduled Runs​

When you adjust a subgraph and that change should affect what a Job refreshes, commit the Job (together with your other Project changes) to version control, then deploy to the environment. Scheduled runs use the deployed Job definition. Until you commit and deploy, the Coalesce Scheduler keeps running the last deployed version.

For how subgraphs relate to Jobs and selector queries, see Using Subgraphs to Build Your Pipeline. For the commit and deploy sequence, see Creating and Run Jobs.

How to Schedule a Job​

  1. In your Coalesce App, go to the Deploy page.

  2. Next to any Job, click the dropdown and select View Scheduled Jobs. Then click Create Job Schedule.

    A dropdown menu under a job called Test Refresh Job within a development environment. The menu offers options to run the job, create a job schedule, or view existing schedules.

  3. Configuration: Fill out the Job Schedule form. If you click on the Job to create a Job schedule, the Project, Environment, and Job Name will be pre-filled.

    1. Project - You’ll be able to create Jobs for any Project you have Environment Admin access to.
    2. Environment - Select the Environment in the Project.
    3. Job Name - Select an existing Job in the Environment.
    4. Job Schedule Name - Choose a unique name to make it easy to identify later.
    5. Cron Expression - Enter a Cron expression in UTC time for the Job frequency.
    6. Timezone - Set the timezone for Job runs.
    7. Retry on Failure - Toggle to retry the entire Job if it fails. This will retry the entire Job. Retries apply only to runs that start and then fail. A run that's rejected before it starts, for example because the Job Schedule's access token was revoked, isn't retried.
      1. Number of Retries - Enter the number of times a Job will attempt to run if it fails the first time.
      2. Time Between Retries - Enter the number of minutes between each retry.
      3. Retry Job From - Beginning will run the entire Job again. Point of Failure will retry the Job from when it failed.
    This image shows a web interface for editing a job schedule in the Coalesce platform. The interface includes various configuration options such as project, environment, job name, schedule frequency, and retry settings for managing automated jobs.

  4. Notifications: Coalesce will send an email notification for successful, retries, and failed Jobs.

    1. Email on: Choose if you want emails to be sent for Success only, Errors only (includes retries), or both.
    2. Email Addresses: Each email address should be entered on a new line.
    Editing a job schedule in a development environment. It includes fields for the project, environment, job name, and job schedule name, along with a cron expression for scheduling

  5. Parameters - Enter runtime parameters as JSON for each run this schedule triggers. If the Environment has default parameters, they appear in the editor and you can override them for this schedule. Coalesce validates the JSON the same way as in the Deploy Wizard. See Parameters for format and usage in templates.

    The Parameters step of the Create Job Schedule wizard in Coalesce. Override environment is enabled and the Parameters field shows JSON that replaces environment defaults for this schedule.

  6. Save the Job. You’ll be able to see the Job on the Job Schedules page.

Detailed view of a specific job schedule within a development environment, including the last execution time, next execution time, and the run status marked as Successful.
Refresh the Page

Refresh the page to see the status of a Scheduled Job.

View Scheduled Jobs​

You can view Scheduled Jobs in two ways.

  1. Select View Schedules next to the Job you want to view the schedules for.

    A dropdown menu under a job called Test Refresh Job within a development environment. The menu offers options to run the job, create a job schedule, or view existing schedules.

  2. Select View Schedules Jobs next to Deploy to see all Scheduled Jobs for the Environment.

A dropdown menu next to the deploy button offers shows options for environment settings, viewing scheduled jobs, and documentation

You’ll be able to view a list of Scheduled Jobs on the Jobs Schedule page.

Job schedules within a development environment, showing details like project name, environment, job name, schedule name, last execution, next execution, and run status. The list shows three different job schedules with their respective statuses and times

Edit Scheduled Jobs​

  1. On the Jobs Schedule page, in the Actions column, select Edit.
  2. You can edit the Job Schedule Name, Cron Expression, and Parameters. If you need to edit the Project, Environment, or Job, create a new Job Schedule.
Editing a job schedule in a development environment. It includes fields for the project, environment, job name, and job schedule name, along with a cron expression for scheduling
Editing Job Schedules Saved by Another User

When you open a Job Schedule that someone else last saved, Coalesce shows a notice that saving it will run it with your credentials. After you select Save, the Job Schedule runs as you, with your Coalesce access token and your data platform credentials for that Environment.

Scheduled Jobs Notifications and Alerts​

Notifications can be sent using email. You can configure emails to be sent for Success, Error, or Both.

You can also edit the notifications at any time by editing the Job Schedule.

Editing job schedule notifications, allowing the user to configure email notifications for job successes, errors, or both. It includes fields to input multiple email addresses.

Notifications are sent for:

  • Each successful Job Schedule.
  • Each failed Job Schedule.

Pause or Resume Scheduled Jobs​

If you need to stop a schedule without removing it, pause it instead of deleting the Job Schedule.

  1. On the Jobs Schedule page, in the Actions column, select Pause.
  2. To start runs again, select Resume from the same menu.

While a schedule is paused, Next Execution shows Paused. The Job does not run until you resume.

Recover a Paused Job Schedule​

A Job Schedule pauses itself when 3 runs in a row are rejected as they try to start, because of an authentication or permission error. When this happens, Next Execution shows Paused even though no user paused it. If email notifications are on for the Job Schedule, its notification list may also get an email titled Schedule Paused After Repeated Authentication Failures.

Each Job Schedule runs as the user who last saved it, with that user's access token. Runs are rejected when:

  • That user's access tokens were revoked, for example because the user was disabled or deleted, or changed their password before your organization moved to the Authentication Upgrade. See Access Token Revoked.
  • That user was removed from your organization. Their access token still works, but they no longer have a role in your organization.

Selecting Resume on its own doesn't fix this. Resume keeps the same access token, so the next run is rejected and the Job Schedule pauses again after 1 failed run. Updating credentials in the Environment doesn't help either, because it doesn't replace the access token saved with the Job Schedule.

To recover a paused Job Schedule, you need to be an Org Admin, a Project Admin for the Project, or an Environment Admin, and you need your own data platform credentials saved in the Environment. If you haven't added them, Coalesce asks you to add them in the Environment settings before you can save.

  1. On the Jobs Schedule page, in the Actions column, select Edit.
  2. Select Save. You don't need to change anything.
  3. In the Actions column, select Resume. Saving alone leaves the Job Schedule paused.
  4. Repeat these steps for each paused Job Schedule.

Saving stores a fresh access token and makes you the user the Job Schedule runs as, so it runs with your Coalesce and data platform credentials from then on. You don't need to re-create the Job Schedule, even if the person who last saved it has left your organization. To keep production Job Schedules from depending on one person's account, run them as a service account. Any Org Admin, Project Admin, or Environment Admin who saves a Job Schedule later takes it over.

Delete Job Schedules​

To delete a Job Schedule, go to the Jobs Schedule page, in the Actions column, select Delete. Pausing is a better option when you plan to use the same schedule again soon.

  • You can delete a Scheduled Job at any time.
Deleting a Job

If you delete a Job attached to a schedule, then commit the deleted Job change to the deploy branch, then try to deploy, you’ll get a warning message in the Review Plan step. The Jobs Schedules for the deleted Job will be removed at the next scheduled run and won’t be part of the deploy.

A deployment review screen with a warning message stating that certain job schedules will be deleted and will not run after deployment. A specific job schedule named Test delete is highlighted as being marked for deletion

Scheduled Job Status and Results​

On the Job Schedule page, you can see:

  • Last Execution - Last time the Scheduled Job was run displayed in system time.
  • Next Execution - Next Scheduled Job run time, or Paused when the schedule is paused.
  • Run Status - Most recent status of the Scheduled Job.
    • Successful: The Job ran without issue.
    • Running: Job is currently running.
    • Failed: The Job failed to run.
    • Test failed: The Job ran, but the Node tests failed.

To view the results click on the Run Status of any scheduled job.

Results of a specific job schedule within the development environment, including details such as the project, environment, job name, schedule name, last execution, next execution, and run status, with a successful execution indicated. The successful text is higlighted

Scheduler Permissions​

Important Scheduling Information​

  • You can create multiple schedules for each Job.
  • Refreshed All Nodes and Ad-Hoc jobs aren’t part of the Job Scheduler and can only be run using the CLI or API.
  • Schedules use Cron to run and they are in UTC time.
  • Selecting Run Job from the Jobs menu will run the Job one-time. It will not change the Scheduled Job.
  • You can pause a Scheduled Job from the Actions column on the Jobs Schedule page instead of deleting it or changing the cron expression.
  • Scheduled Jobs run in parallel. If more than one Job is scheduled for the same time, they will run at the same time. If there is an ad hoc refresh during a Scheduled Job, they will run at the same time.
  • Scheduled Jobs aren’t committed to your Git provider.
  • The Coalesce Scheduler only runs deployed Jobs. Subgraph or Job changes in the Workspace do not affect scheduled runs until you commit, deploy, and the environment has the updated definition.
  • Each Job Schedule can define its own parameter JSON so you can run the same deployed Job on different schedules with different runtime values without creating duplicate Environments.

Resources​