citusdata / citusdata/pg_cron

Run multiple instances of a job

Open
#141 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
C
Stars
3.9k
Forks
260
Avg merge
5d 5h
Merged PRs (30d)
1

Description

I have a task that is scheduled to run every minute. Sometimes, this task can take longer than 1 minute to run, and in that event I'd like to start a second instance of the same task running in parallel (for computational efficiency reasons). Unfortunately, this doesn't seem to be supported. From the docs:

> pg_cron can run multiple jobs in parallel, but it runs at most one instance of a job at a time. If a second run is supposed to start before the first one finishes, then the second run is queued and started as soon as the first run completes.

I can think of several ways to simulate this behavior (e.g. by adding a second "dummy" instance that has internal logic to only do the task if the other instance isn't running), but this would be very cumbersome. What I'd ideally like is to be able to specify a maximum number of parallel instances that can run per task. By setting this to 1, you'd get the current behavior, be we could set this to a higher number to enable more parallelism for certain tasks. The purpose for having the user controlled limit, is to prevent the cron tasks from accidentally consuming all the resources on the server.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the documentation passage describing pg_cron’s one-instance-at-a-time behavior, then trace the scheduler and job execution paths that enforce it. Define how a per-job maximum parallel instance count would work, including the current behavior when the limit is 1, and verify that the limit prevents unintended resource exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.