cockroachdb / cockroachdb/cockroach
schedules_BACKUP_last_completed_time resets on crdb restart
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The schedules_BACKUP_last_completed_time metric resets on crdb restart. After a rolling restart of all crdb nodes, this metric appears to indicate that no backups have been taken.
**To Reproduce**
Take a backup, restart all crdb pods, and query schedules_BACKUP_last_completed_time; it will report a value of 0 until the next backup succeeds.
**Expected behavior**
schedules_BACKUP_last_completed_time should report the most recent backup success timestamp even after restarting pods.
To fix, we propose that crdb populates this metric on startup, as well as updating on successful backups. If this behavior is acceptable, we can try implementing it ourselves if that would be helpful.
h/t @dasrirez @DuskEagle
Jira issue: CRDB-24392
Contributor guide
Assessment
This issue has not been assessed yet.