pingcap / pingcap/tidb

ddl: old owner silently drops tidb_max_dist_task_nodes in mixed-version distributed backfill

Open
#67,015 6 comments 0 reactions 1 assignee Claimed by @D3Hunter View on GitHub
component/ddl component/DXF impact/upgrade severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This issue happens on the mixed-version split:
- new submitter: `v8.5.6`
- old DDL owner / dist-task scheduler: `v8.5.5`

A reliable runtime path is a partitioned add-index backfill task.

1. Prepare a mixed-version cluster with one old TiDB node (`v8.5.5`) and one new TiDB node (`v8.5.6`).
2. Make sure the new node is the DDL owner first, then verify the control path:
```sql
create database if not exists compat_66376;
use compat_66376;
create table t(
a bigint auto_random primary key,
b int
) partition by hash(a) partitions 20;
insert into t values
(1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8),(9,9),(10,10),
(11,11),(12,12),(13,13),(14,14),(15,15),(16,16),(17,17),(18,18),(19,19),(20,20),
(21,21),(22,22),(23,23),(24,24),(25,25),(26,26),(27,27),(28,28),(29,29),(30,30);
split table t between (3) and (8646911284551352360) regions 50;
set @@session.tidb_max_dist_task_nodes=1;
alter table t add index idx_b(b);
```
On the new-owner path, the task row keeps `max_node_count=1` and all subtasks stay on one executor.
3. Now force the old node to become DDL owner by disabling DDL on the new node:
```sql
set @@global.tidb_enable_ddl = false;
```
4. From the new `v8.5.6` node, run the same task again with:
```sql
set @@session.tidb_max_dist_task_nodes=1;
alter table t add index idx_b(b);
```
5. Inspect:
- `mysql.tidb_global_task_history.max_node_count`
- `meta.job.reorg_meta.max_node_count`
- `mysql.tidb_background_subtask_history`
- owner log lines about `eligible instances`

### 2. What did you expect to see? (Required)

I expected the user-configured node cap to be honored even when the task is submitted from a new node while an old node is the DDL owner / dist-task scheduler.

Concretely:
- the persisted task row should keep `max_node_count=1`
- `meta.job.reorg_meta.max_node_count` should also keep `1`
- the task should stay on one executor

### 3. What did you see instead (Required)

On the mixed-version path `new submitter (v8.5.6) -> old owner/scheduler (v8.5.5)`:
- the persisted task row fell back to `max_node_count=0`
- `meta.job.reorg_meta.max_node_count` was omitted
- the task still fanned out across both TiDB executors (`10/10` subtasks)
- old owner log still reported `eligible instances ... num=2`

Control under the new-owner path behaved correctly:
- `max_node_count=1`
- all 20 subtasks stayed on one executor

So the new node-cap semantic is silently dropped when the old owner materializes the distributed task.

### 4. What is your TiDB version? (Required)

Mixed-version reproduction:

Old TiDB node:
```text
Release Version: v8.5.5
```

New TiDB node:
```text
Release Version: v8.5.6
```

### Additional context

This is a patch-compatibility / rolling-upgrade issue rather than a same-version functional bug.

The user-facing impact is that `tidb_max_dist_task_nodes` becomes effectively ignored in the `new submitter -> old owner/scheduler` path, so a task that was explicitly capped to one node may still fan out across multiple executors.

Reproduction was validated in a cloud mixed-version environment. A detailed local report artifact is available on my side if needed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.