pingcap / pingcap/tidb

materialized view scheduler should surface operator-facing diagnostics when clock skew pushes next_time into the future

Open
#67,671 0 comments 0 reactions 0 assignees View on GitHub
component/mview feature/developing type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

When MV refresh/purge jobs finish, the scheduler writes the next schedule time back into `mysql.tidb_mview_refresh_info` / `mysql.tidb_mlog_purge_info`. If the job completes while the local TiDB wall clock has jumped forward, the persisted `next_time` can also be pushed far into the future. After clocks are corrected, refresh/purge can remain quiet until that future `next_time` arrives.

This looks like a design limitation / operability gap rather than a correctness bug by itself. The main problem is that the operator-facing signal is weak: from the outside it can look like refresh/purge stopped progressing, but the real reason is that the scheduler is waiting on a future-pinned `next_time`. In practice the likely recovery path is operator intervention, not panic or automatic self-healing.

Requested enhancement:
- Emit a clear warning or critical log when MV refresh or purge `next_time` is far ahead of the current TiDB wall clock.
- Expose a metric or status signal so operators can detect this condition without manually inspecting internal system tables.
- Document the expected behavior and recommended operator action after clock skew or a large time jump.
- If a stronger time source such as PD/TSO is used for detection, use it for diagnostics or alerting only. Avoid panic or forced shutdown as the default behavior.

Why this matters:
- Long periods without refresh/purge progress can be misread as a product stall even though the scheduler is simply waiting on a future-pinned `next_time`.
- Better observability would make it much easier for operators to identify the condition and intervene safely.

Background:
- Observed in April 2026 during validation of materialized view scheduling under clock skew / time jump scenarios.
- Exact `SELECT tidb_version()` output was not preserved before cleanup.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.