materialized view scheduler should surface operator-facing diagnostics when clock skew pushes next_time into the future
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
When MV refresh/purge jobs finish, the scheduler writes the next schedule time back into `mysql.tidb_mview_refresh_info` / `mysql.tidb_mlog_purge_info`. If the job completes while the local TiDB wall clock has jumped forward, the persisted `next_time` can also be pushed far into the future. After clocks are corrected, refresh/purge can remain quiet until that future `next_time` arrives.
This looks like a design limitation / operability gap rather than a correctness bug by itself. The main problem is that the operator-facing signal is weak: from the outside it can look like refresh/purge stopped progressing, but the real reason is that the scheduler is waiting on a future-pinned `next_time`. In practice the likely recovery path is operator intervention, not panic or automatic self-healing.
Requested enhancement:
- Emit a clear warning or critical log when MV refresh or purge `next_time` is far ahead of the current TiDB wall clock.
- Expose a metric or status signal so operators can detect this condition without manually inspecting internal system tables.
- Document the expected behavior and recommended operator action after clock skew or a large time jump.
- If a stronger time source such as PD/TSO is used for detection, use it for diagnostics or alerting only. Avoid panic or forced shutdown as the default behavior.
Why this matters:
- Long periods without refresh/purge progress can be misread as a product stall even though the scheduler is simply waiting on a future-pinned `next_time`.
- Better observability would make it much easier for operators to identify the condition and intervene safely.
Background:
- Observed in April 2026 during validation of materialized view scheduling under clock skew / time jump scenarios.
- Exact `SELECT tidb_version()` output was not preserved before cleanup.
Contributor guide
Assessment
This issue has not been assessed yet.