humanmade / humanmade/Cavalcade

Impending DST cutover caused jobs to run continuously

Open
#74 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
PHP
Stars
542
Forks
48
PR merge metrics
No merged PRs in 30d

Description

Last evening, at exactly 9 PM EDT (2 AM GMT), hourly jobs started running continuously--instead of being scheduled for the next hour. This caused a major load event and we had to stop Cavalcade everywhere. Sample log showing the issue:

```
20413 wp_privacy_delete_old_export_files 2018-11-03 19:14:32 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 20:14:32 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:34 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:36 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:37 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:39 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:40 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:41 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:42 completed
```

This morning, it _seems_ like we can get cavalcade happy by running `wp site list --field=url | xargs -n 1 -I % wp cron event run --due-now --url=%` to bring all jobs into the present. The WordPress site in question uses UTC internally; the host servers (both application and database) are on EDT/EST.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.