Schedule silently stops firing - scheduler workflow keeps a pending timer with no corresponding physical timer task

Temporal version

• Temporal Server: 1.31.0
• Persistence: PostgreSQL (postgres12_pgx)
• Visibility: Elasticsearch
• numHistoryShards: 512
• Scheduler implementation: v1, workflow-backed (temporal-sys-scheduler-workflow, TemporalNamespaceDivision = "TemporalScheduler")
• ~32 schedules across 14 namespaces

Summary

A schedule stops firing permanently and silently. The internal scheduler workflow still holds a pending timer in its mutable state, but no
corresponding row exists in the timer_tasks table, so the timer never fires. There is no error anywhere: no failed workflow task, no DLQ entry, no
retry counter, no log line, no metric.

schedule describe keeps reporting a normal-looking NextRunTime, which makes the failure effectively invisible — the scheduler still computes its next action times, it simply never wakes up to act on them.

tdbg workflow refresh-tasks recovers the schedule, which confirms the logical timer was intact and only the physical task was missing.

Evidence

Scheduler workflow history for a daily schedule :

1250  2026-08-27T00:01:31Z  WORKFLOW_TASK_TIMED_OUT     <- last normal daily cycle
                                                           (2026-08-28 00:00 never happened)
1261  2026-08-28T18:05:25Z  MARKER_RECORDED
1262  2026-08-28T18:05:25Z  MARKER_RECORDED
1263  2026-08-28T18:05:25Z  MARKER_RECORDED
1264  2026-08-28T18:05:25Z  WORKFLOW_PROPERTIES_MODIFIED
1265  2026-08-28T18:05:25Z  TIMER_STARTED               <- timer armed, never fires
1266  2026-08-28T23:39:22Z  WORKFLOW_EXECUTION_SIGNALED <- manual unpause
1267  2026-08-28T23:39:22Z  WORKFLOW_TASK_SCHEDULED
1268  2026-08-28T23:39:22Z  WORKFLOW_TASK_STARTED
1269  2026-08-28T23:39:32Z  WORKFLOW_TASK_TIMED_OUT
1270  2026-08-28T23:39:32Z  WORKFLOW_TASK_SCHEDULED
1271  2026-08-28T23:39:32Z  WORKFLOW_TASK_STARTED
1272  2026-08-28T23:39:37Z  WORKFLOW_TASK_COMPLETED
1273  2026-08-28T23:39:37Z  MARKER_RECORDED
1274  2026-08-28T23:39:37Z  WORKFLOW_PROPERTIES_MODIFIED
                            (nothing for 2 days 13 hours)
1275  2026-08-31T07:40:19Z  TIMER_FIRED                 <- fires immediately after refresh-tasks
1276  2026-08-31T07:40:19Z  TIMER_FIRED
...
1285  2026-08-31T07:40:32Z  TIMER_STARTED               <- normal operation resumes

The timer started at event 1265 sat pending for 2 days 13 hours, then fired
within seconds of refresh-tasks being issued. It was present in mutable
state the whole time; only its physical task was missing.

During the same window, querying persistence directly returned no timer task
for that workflow:

SELECT visibility_timestamp, task_id
FROM timer_tasks
WHERE shard_id = <shard>
  AND position(convert_to('temporal-sys-scheduler:xxx-daily','UTF8') IN data) > 0;
-- 0 rows while stuck, 1 row after refresh-tasks

Recovery

tdbg --address <frontend>:7233 --namespace <ns> \
  workflow refresh-tasks \
  --workflow-id "temporal-sys-scheduler:<schedule-id>"

Confirmed working on a genuinely stuck v1 scheduler: 0 timer tasks before,
1 after, schedule resumed on its normal cadence. This is non-destructive and
preserves history, workflow ID and input, unlike delete + recreate.

Delete + recreate also recovers the schedule, but is not durable : one of our
schedules was recreated and stopped again within 24 hours.

What does NOT recover it

• Pause / unpause: the signal is accepted into history and processed, but the
  schedule does not resume (see events 1266-1274 above).
• Restarting history, frontend and worker pods: the state is persisted,
  not in memory.

Possibly relevant: routine workflow task timeouts

The scheduler workflow times out on essentially every daily execution since
2026-06-25 : around 120 occurrences, consistently at 00:00:1x, i.e. 10s
(WorkflowTaskTimeout) after the midnight trigger:

1237  2026-08-26T00:00:17Z  WORKFLOW_TASK_TIMED_OUT
1240  2026-08-26T00:00:30Z  WORKFLOW_TASK_TIMED_OUT
1250  2026-08-27T00:01:31Z  WORKFLOW_TASK_TIMED_OUT

Each of these recovers normally, so timeouts alone are clearly not sufficient
to trigger the failure. We report it because it shows the per-namespace system
worker routinely fails to process scheduler workflow tasks within 10s when many
daily schedules fire simultaneously at midnight, and this may be the condition
under which the physical timer task creation is lost.

We have not been able to identify the exact trigger.

Environment context

The first occurrences coincided with persistence contention on our cluster
(SQL connection pool saturated at the default 20 connections per history pod,
serviceerror_Unavailable peaking around 500/s). We have since raised the pool
and scaled the history service, and the contention is gone — but schedules kept
dying afterwards, so contention looks like an amplifier rather than the cause.

Related

• #10579 describes the same user-visible symptoms (schedule stops firing,
  pause/unpause does not recover) on the same version and persistence stack.
  However its documented trigger requires a workflow-level retry policy
  producing a multi-run chain, and all our schedules have RetryPolicy: null.
  We also find no trace of WatchWorkflow: last event did not have correct attrs in worker logs (7 days) nor in the scheduler workflow history. So this
  may be a different path to a similar end state.
• #11791 / #11432 discuss pure-task handling in the CHASM scheduler. Our
  schedules are v1 workflow-backed, so those do not apply directly, though the
  end state described in #11432 (execution silently stops receiving timer
  processing, no self-healing path) is very close to what we observe.

Detection

We detect stuck schedulers by looking for scheduler workflows with no timer
task in persistence. Note this is a partial signal: it produces false
positives for paused schedules, and we have also seen at least one stuck
scheduler that did have unrelated timer rows present, so it under-reports.

WITH sched AS (
    SELECT ce.shard_id, ce.namespace_id, ce.workflow_id
    FROM current_executions ce
    WHERE ce.workflow_id LIKE 'temporal-sys-scheduler:%'
)
SELECT n.name AS namespace,
       replace(s.workflow_id, 'temporal-sys-scheduler:', '') AS schedule_id,
       s.shard_id
FROM sched s
LEFT JOIN namespaces n ON n.id = s.namespace_id
WHERE NOT EXISTS (
    SELECT 1 FROM timer_tasks t
    WHERE t.shard_id = s.shard_id
      AND t.visibility_timestamp > now()
      AND position(convert_to(s.workflow_id, 'UTF8') IN t.data) > 0
)
ORDER BY n.name, schedule_id;

(timer_tasks has no workflow_id column, hence the search inside the raw
proto blob.)

Happy to provide full histories, shard IDs, mutable state dumps, or run any
diagnostic that would help narrow this down.

Start with the v1 workflow-backed scheduler and compare its pending timer state with rows in the PostgreSQL timer_tasks table. Reproduce the missing physical task around workflow-task timeouts or persistence contention, using workflow history and the refresh-tasks command as evidence. Done means identifying the loss path and preventing a pending logical timer from remaining without a physical task.