pingcap / pingcap/tidb

COMPARE MATERIALIZED VIEW cannot be resource-controlled for large MViews and hits TiFlash memory limit

Open
#69,515 0 comments 0 reactions 0 assignees View on GitHub
component/mview feature/developing severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

On a TiDB build with materialized view support, prepare a large materialized view backed by TiFlash. The observed case used two `REFRESH FAST` materialized views with TiFlash replicas available:

- `db.mv_large_1`: about 419M rows in the materialized view, about 58 GiB reported by `information_schema.tables.data_length`.
- `db.mv_large_2`: about 292M rows in the materialized view, about 40 GiB reported by `information_schema.tables.data_length`.

The MView definition is an aggregate over a large base table with `GROUP BY` columns and aggregate columns such as `COUNT`, `SUM`, `MAX`, and `MIN`. The latest `mysql.tidb_mview_refresh_info.LAST_SUCCESS_READ_TSO` was positive and recent, and automatic refresh jobs were still succeeding.

Run `COMPARE MATERIALIZED VIEW ... AS OF TIMESTAMP` at the latest successful read TSO:

```sql
SELECT t.table_name,
r.LAST_SUCCESS_READ_TSO,
TIDB_PARSE_TSO(r.LAST_SUCCESS_READ_TSO) AS last_success_time
FROM information_schema.tables t
JOIN mysql.tidb_mview_refresh_info r ON r.MVIEW_ID = t.TIDB_TABLE_ID
WHERE t.table_schema = 'db'
AND t.table_name = 'mv_large_2';

COMPARE MATERIALIZED VIEW db.mv_large_2
AS OF TIMESTAMP TIDB_PARSE_TSO();
```

The same failure is still reproducible after trying to bound the join and TiFlash execution with session variables, for example:

```sql
SET SESSION max_execution_time = 1800000;
SET SESSION tidb_mem_quota_query = 1073741824;
SET SESSION tidb_mv_maintain_mem_quota = 1073741824;
SET SESSION tidb_mv_maintain_isolation_read_engines = 'tiflash';
SET SESSION tidb_allow_mpp = ON;
SET SESSION tidb_enforce_mpp = ON;
SET SESSION tidb_isolation_read_engines = 'tiflash';
SET SESSION tiflash_fine_grained_shuffle_stream_count = 1;
SET SESSION tidb_max_tiflash_threads = 1;
SET SESSION tidb_opt_tiflash_concurrency_factor = 1;
SET SESSION tidb_distsql_scan_concurrency = 1;
SET SESSION tiflash_mem_quota_query_per_node = 10737418240;
SET SESSION tidb_max_bytes_before_tiflash_external_join = 67108864;
SET SESSION tidb_hash_join_concurrency = 1;
SET SESSION tidb_merge_join_concurrency = 1;

COMPARE MATERIALIZED VIEW db.mv_large_2
AS OF TIMESTAMP TIDB_PARSE_TSO();
```

Also tried disabling hash join and enabling index merge join for the same statement:

```sql
SET SESSION tidb_opt_enable_hash_join = OFF;
SET SESSION tidb_enable_index_merge_join = ON;
SET SESSION tidb_max_bytes_before_tiflash_external_join = 67108864;
SET SESSION tidb_merge_join_concurrency = 1;

COMPARE MATERIALIZED VIEW db.mv_large_2
AS OF TIMESTAMP TIDB_PARSE_TSO();
```

Observed attempts on 2026-06-29:

- External join spill threshold `1 GiB`, hash join concurrency `1`, TiFlash stream count `4`: failed after about 30.6s.
- External join spill threshold `64 MiB`, TiFlash stream count `1`, TiFlash threads `1`: failed after about 28.4s.
- `tidb_opt_enable_hash_join=OFF`, `tidb_enable_index_merge_join=ON`, external join spill threshold `64 MiB`: failed after about 28.5s.

Additional observability gap:

```sql
EXPLAIN COMPARE MATERIALIZED VIEW db.mv_large_2 AS OF TIMESTAMP ...;
```

returns a syntax error, and `information_schema.statements_summary` records `STMT_TYPE = CompareMaterializedView` but the `PLAN` field is empty. This makes it hard to verify whether the internal compare query honors join/spill/merge settings.

### 2. What did you expect to see? (Required)

`COMPARE MATERIALIZED VIEW ... AS OF TIMESTAMP` should be usable on large materialized views in a controlled way.

At minimum, one of the following should be true:

- The internal compare join honors TiFlash external join spill / join algorithm / concurrency controls and can complete under bounded memory.
- The statement exposes a documented way to run the compare in bounded chunks or with a resource cap.
- The statement provides an inspectable plan or actionable diagnostics showing which internal join path is used and why spill or merge join is not selected.

The statement should not require effectively unbounded TiFlash memory for large but valid MView objects.

### 3. What did you see instead (Required)

The statement consistently failed with a TiFlash MPP memory error. Lowering join spill thresholds, reducing TiFlash concurrency, and disabling hash join from the TiDB session did not make the full compare complete.

Representative error:

```text
ERROR 1105 (HY000): other error for mpp stream: Code: 0, e.displayText() = DB::TiFlashException: Memory limit (total) exceeded caused by 'RSS(Resident Set Size) much larger than limit' : process memory size would be 52.22 GiB for (attempt to allocate chunk of 1139938 bytes), limit of memory for data computing : 51.20 GiB. Memory Usage of Storage: non-query: peak=1.07 GiB, amount=1.84 MiB; kvstore: peak=61.71 MiB, amount=0.00 B; query-storage-task: peak=385.02 MiB, amount=47.18 MiB; fetch-pages: peak=0.00 B, amount=0.00 B; ...
```

The failure did not crash TiDB or TiFlash in the observed run, but it means the current `COMPARE MATERIALIZED VIEW` implementation is not practical for these large MViews and does not appear to expose an effective resource-control path.

### 4. What is your TiDB version? (Required)

```text
Release Version: v8.5.4-feature.release-8.5-materialized-view
Edition: Enterprise
Git Commit Hash: 8aff9064f372a651647de33f3953bc448af6b8f0
Git Branch: HEAD
UTC Build Time: 2026-06-29 05:56:05
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: tikv
Enterprise Extension Commit Hash: 7d43ff65ebc145bd63fa84cb368f8775be906998
```

TiFlash version observed in `information_schema.cluster_info`:

```text
8.5.4-20260409-5d56de0
Git hash: 5d56de051244ac7c893de5ac029f0013098f95dc
```

Contributor guide

Open the contributing guide

Research direction

Start with the COMPARE MATERIALIZED VIEW execution path, reproduce against the large TiFlash-backed views, and inspect the statement-summary behavior; the issue names no source file or test. Done means bounded-memory execution or a documented resource cap, plus an inspectable plan or actionable diagnostics, without the reported TiFlash memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.