pingcap / pingcap/tidb

MView: MLOG auto analyze is much more frequent than base/MView and stats health stays near zero

Open
#69,723 3 comments 0 reactions 0 assignees View on GitHub
component/mview feature/developing severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. Enable materialized view log auto analyze and keep the default predicate-column analyze behavior:

```sql
SHOW VARIABLES WHERE Variable_name IN (
'tidb_enable_auto_analyze',
'tidb_auto_analyze_ratio',
'tidb_mlog_auto_analyze_ratio',
'tidb_analyze_column_options'
);

-- Observed values:
-- tidb_enable_auto_analyze = ON
-- tidb_auto_analyze_ratio = 0.1
-- tidb_mlog_auto_analyze_ratio = 10
-- tidb_analyze_column_options = PREDICATE
```

2. Create a workload with many base tables, materialized views, and corresponding MLOG tables. In the observed run there were:

```text
base tables: 1204 total, including 1201 regular workload base tables
materialized views: 1205
MLOG tables: 1201
```

3. Keep running a write workload on the base tables and refresh/purge the materialized views continuously, so the MLOG tables receive new rows and consumed MLOG rows are purged.

4. After updating TiDB to the version below, observe auto analyze jobs from `2026-07-06 09:47:06 UTC` to `2026-07-08 06:23:26 UTC`:

```sql
SET @since := TIMESTAMP('2026-07-06 09:47:06');

SELECT
CASE
WHEN table_name REGEXP '^\\$mlog\\$' THEN 'mlog'
WHEN table_name REGEXP '^bet_mview' THEN 'mview'
ELSE 'base'
END AS table_type,
COUNT(*) AS finished_jobs,
ROUND(COUNT(*) / (TIMESTAMPDIFF(SECOND, @since, NOW()) / 3600), 2) AS jobs_per_hour,
COUNT(DISTINCT CONCAT(table_schema, '.', table_name)) AS distinct_tables,
SUM(processed_rows) AS processed_rows,
ROUND(SUM(processed_rows) / NULLIF(COUNT(*), 0), 2) AS avg_processed_rows_per_job,
MAX(end_time) AS last_end
FROM mysql.analyze_jobs
WHERE start_time >= @since
AND state = 'finished'
GROUP BY table_type
ORDER BY table_type;
```

5. Check stats health by table type:

```sql
SHOW STATS_HEALTHY WHERE Db_name = '';
```

Also check the `modify_count / count` distribution for MLOG tables:

```sql
WITH mlog AS (
SELECT tidb_table_id AS table_id
FROM information_schema.tables
WHERE table_schema = ''
AND table_name REGEXP '^\\$mlog\\$'
), s AS (
SELECT
m.table_id,
sm.count AS stats_count,
sm.modify_count,
CASE WHEN sm.count = 0 THEN NULL ELSE sm.modify_count / sm.count END AS modify_ratio
FROM mlog m
LEFT JOIN mysql.stats_meta sm ON sm.table_id = m.table_id
)
SELECT
COUNT(*) AS mlog_tables,
SUM(stats_count = 0 AND modify_count > 0) AS count_zero_modify_positive,
SUM(modify_ratio >= 0.1) AS ratio_ge_0_1,
SUM(modify_ratio >= 1) AS ratio_ge_1,
SUM(modify_ratio >= 10) AS ratio_ge_10,
SUM(modify_ratio >= 100) AS ratio_ge_100,
ROUND(AVG(modify_ratio), 2) AS avg_modify_ratio,
MAX(modify_ratio) AS max_modify_ratio
FROM s;
```

### 2. What did you expect to see? (Required)

MLOG auto analyze should not dominate the analyze workload by an order of magnitude compared with base tables and materialized views under a normal refresh/purge workload.

After MLOG rows are purged and MLOG analyze jobs finish successfully, MLOG table stats health should recover or at least stay in a reasonable range. MLOG tables should not remain almost permanently unhealthy and repeatedly become auto-analyze candidates.

### 3. What did you see instead (Required)

From `2026-07-06 09:47:06 UTC` to `2026-07-08 06:23:26 UTC`, about `44.61` hours after the TiDB image update:

The observed MLOG-specific ratio setting was:

```text
tidb_mlog_auto_analyze_ratio = 10
```

```text
table_type finished_jobs jobs_per_hour distinct_tables processed_rows avg_processed_rows_per_job
base 351 7.87 351 2,491,948,998 7,099,569.79
mlog 19,478 436.67 1,200 6,622,843 340.02
mview 1,200 26.90 1,200 7,885,538,068 6,571,281.72
```

In the latest one-hour window, MLOG still dominated the analyze jobs:

```text
table_type finished_jobs distinct_tables processed_rows
base 1 1 10,161,666
mlog 348 343 132,920
mview 1 1 9,281,676
```

Stats health was normal for most base tables and materialized views, but almost all MLOG tables had health close to zero:

```text
table_type tables min_health avg_health max_health health_0 health_lt_90 health_ge_90
base 1204 0 92.77 99 2 2 1202
mview 1205 0 93.87 99 4 4 1201
mlog 1201 0 0.11 67 1198 1201 0
```

The MLOG `modify_count / count` distribution also stayed abnormal:

```text
mlog_tables = 1201
count_zero_modify_positive = 6
modify_ratio >= 0.1 = 1195
modify_ratio >= 1 = 1193
modify_ratio >= 10 = 1170
modify_ratio >= 100 = 331
avg_modify_ratio = 83.62
max_modify_ratio = 2746
```

The current MLOG analyze jobs do collect column histograms for most MLOG tables, so this is different from the symptom where finished MLOG analyze jobs produced no usable column histograms:

```text
MLOG tables with column histograms: 1200 / 1201
histogram columns per analyzed MLOG table: up to 8
```

Related issues:

- https://github.com/pingcap/tidb/issues/69574
- https://github.com/pingcap/tidb/issues/69575

### 4. What is your TiDB version? (Required)

```text
Release Version: v8.5.4-feature.release-8.5-materialized-view
Edition: Enterprise
Git Commit Hash: 19544ab81d3472abf4219aa9d658f8a90eebbf90
Git Branch: HEAD
UTC Build Time: 2026-07-06 09:19:48
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: tikv
Enterprise Extension Commit Hash: 7d43ff65ebc145bd63fa84cb368f8775be906998
```

`SELECT VERSION()`:

```text
8.0.11-TiDB-v8.5.4-feature.release-8.5-materialized-view
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the refresh and purge workload, then compare MLOG, base-table, and materialized-view results in mysql.analyze_jobs and SHOW STATS_HEALTHY. Inspect the auto-analyze behavior associated with tidb_mlog_auto_analyze_ratio and mysql.stats_meta, using the provided modify_count/count queries. Done means MLOG jobs no longer dominate and MLOG statistics health recovers or remains reasonable after successful analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.