cockroachdb / cockroachdb/cockroach

schedules_BACKUP_last_completed_time resets on crdb restart

Open
#96,888 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The schedules_BACKUP_last_completed_time metric resets on crdb restart. After a rolling restart of all crdb nodes, this metric appears to indicate that no backups have been taken.

**To Reproduce**

Take a backup, restart all crdb pods, and query schedules_BACKUP_last_completed_time; it will report a value of 0 until the next backup succeeds.

**Expected behavior**

schedules_BACKUP_last_completed_time should report the most recent backup success timestamp even after restarting pods.

To fix, we propose that crdb populates this metric on startup, as well as updating on successful backups. If this behavior is acceptable, we can try implementing it ourselves if that would be helpful.

h/t @dasrirez @DuskEagle

Jira issue: CRDB-24392

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.