Continuously increasing drift in scheduled workflow, more than 4 hours now

Hi,

I have observed a regularly increasing drift in a daily scheduled workflow on GitHub runners. I know that the scheduled time is a minimal time and the workflow starts some time after the scheduled time. However, in the past few months, there is an increasing drift with an average delay of several hours, more than 4 hours now in some cases, depending on the time of the day.

Is it normal, expected, universal? If yes, what is the reason and is there anything planned to improve the situation? If not normal, what could be wrong with that job?

The job has been in place for years. This is the "nightly build" job of an open source project (ref at end of message). It builds and tests the project on Ubuntu and Windows and publishes the binaries in the project's web site. The duration of the job is approximately 1 hour and 40 minutes, except when the repository hasn't been updated in the last 24 hours, in which case it runs just for a few seconds (nothing new to build).

The job was initially scheduled at 00:40 UTC to avoid peak traffic at 00:00. Several years ago, the job used to start within 10-20 minutes. This is just from my recall because the logs are retained for the last 410 days only. From the current logs, in 2025, the average delay was 1h 40mn, already quite a lot, but stable. Since early 2026, the delay is regularly increasing, reaching 4h 30mn now.

This topic was already discussed in https://github.com/orgs/community/discussions/196910 but it seems that it now deserves opening an issue.

Several people expressed similar feelings of increasing delays, although not of the same magnitude of more than 4 hours.

The appropriate time of the day was discussed, suggesting to get away from the "midnight UTC bottleneck", a favorite time for many developers, without relation to their actual time zone. Various times of the day were tested with mixed results, but still more than one or two hours delay:

• 00:40 UTC (initial schedule): from 1h30 to 4h30 delayed over a year.
• 14:37 UTC: 3h delayed.
• 23:11 UTC: 1h to 2h delayed.

Using a Python script, I collect the created and start time of each run over the last 410 saved schedule. See a partial output below. Note that created and start times are always identical and I didn't find any other relevant time in the GitHub API.

Modified schedule at 23:11 UTC:

  Run#   Created               Started               Duration   Status     Origin
------   -------------------   -------------------   --------   --------   ---------
  2376   2026-06-02 01:35:42   2026-06-02 01:35:42   00:00:14   success    schedule
  2375   2026-06-01 00:14:44   2026-06-01 00:14:44   00:00:11   success    schedule
  2374   2026-05-31 00:12:59   2026-05-31 00:12:59   01:25:49   success    schedule

Modified schedule at 14:37 UTC:

  2373   2026-05-29 17:53:36   2026-05-29 17:53:36   01:23:39   success    schedule
  2372   2026-05-28 17:50:14   2026-05-28 17:50:14   01:52:34   success    schedule
  2371   2026-05-27 17:42:39   2026-05-27 20:13:00   00:02:06   failure    schedule
  2370   2026-05-26 17:46:12   2026-05-26 17:46:12   01:47:34   success    schedule

Initial schedule at 00:40 UTC:

  2369   2026-05-25 05:08:22   2026-05-25 05:08:22   00:59:21   failure    schedule
  2368   2026-05-24 04:49:20   2026-05-24 04:49:20   01:50:03   success    schedule
  ...
  2315   2026-04-01 03:52:31   2026-04-01 03:52:31   00:00:12   success    schedule
  ...
  2283   2026-03-01 03:32:43   2026-03-01 03:32:43   00:00:12   success    schedule
  ...
  2255   2026-02-01 03:39:12   2026-02-01 03:39:12   00:00:13   success    schedule
  ...
  2224   2026-01-01 02:57:05   2026-01-01 02:57:05   00:00:14   success    schedule
  ...
  2193   2025-12-01 02:57:02   2025-12-01 02:57:02   01:47:27   success    schedule
  ...
  2160   2025-11-01 02:22:19   2025-11-01 02:22:19   00:00:10   success    schedule
  ...
  2124   2025-10-01 02:21:29   2025-10-01 02:21:29   00:00:14   success    schedule
  ...
  2094   2025-09-01 02:31:11   2025-09-01 02:31:11   00:00:11   success    schedule
  ...
  2061   2025-08-01 03:00:41   2025-08-01 03:00:41   00:00:16   success    schedule
  ...
  2029   2025-07-01 02:39:34   2025-07-01 02:39:34   00:00:11   success    schedule
  ...
  1993   2025-06-01 02:50:41   2025-06-01 02:50:41   01:32:13   success    schedule
  ...
  1962   2025-05-01 02:27:41   2025-05-01 02:27:41   01:30:04   success    schedule
  ...
  1947   2025-04-18 02:12:35   2025-04-18 02:12:35   00:00:35   success    schedule

Project: https://github.com/tsduck/tsduck
Workflow: https://github.com/tsduck/tsduck/blob/master/.github/workflows/nightly-build.yml
Access to workflow runs: https://github.com/tsduck/tsduck/actions/workflows/nightly-build.yml
Python script to get the run times: https://github.com/tsduck/tsduck/blob/master/pkg/github/nightly-build-statistics.py

Start with the linked .github/workflows/nightly-build.yml and pkg/github/nightly-build-statistics.py, then review the recorded workflow runs and GitHub API timestamps. Compare scheduled, created, and started times across the tested UTC schedules. Done means determining whether the increasing delay is expected GitHub Actions behavior or identifying a concrete runner-related failure.