pingcap / pingcap/tidb

The Emergency-level alerts "TiKV_GC_can_not_work" will alert when the cluster is running normally

Open
#37,424 1 comment 0 reactions 0 assignees View on GitHub
sig/transaction type/question
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

When the cluster is running normally and no data is written, the TiKV_GC_can_not_work alarm will occur.

AskTug Reference link:https://asktug.com/t/topic/242796

### 2. What did you expect to see? (Required)

When the cluster is running normally and no data is written, the TiKV_GC_can_not_work alarm will not occur.

### 3. What did you see instead (Required)

When the cluster is running normally and no data is written. View 'sum(increase(tikv_gc_compaction_filter_perform[1d]))' in prometheus is always 0. The second expression condition is met. So it will keep warning.

TiKV_GC_can_not_work:
sum(increase(tikv_gcworker_gc_tasks_vec{task="gc"}[1d])) < 1 and
**(sum(increase(tikv_gc_compaction_filter_perform[1d])) < 1** and
sum(increase(tikv_engine_event_total{db="kv", cf="write", type="compaction"}[1d])) >= 1)

### 4. What is your TiDB version? (Required)

v5.4.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.