pingcap / pingcap/tidb

DDL operation hangs indefinitely due to now DDL owner being in the cluster

Open
#57,047 3 comments 0 reactions 0 assignees View on GitHub
component/ddl severity/minor type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Setting `tidb_enable_ddl` to OFF isn't allowed in a single node cluster. However this safety isn't present when scaling in a cluster.

### 1. Minimal reproduce step (Required)

Start a playground with 2 tidb nodes:
```
tiup playground --db 2 --tiflash 0 --without-monitor v8.3.0
```

Connect to the first node (port 4000) and change `tidb_enable_ddl`:
```
SET GLOBAL tidb_enable_ddl=OFF;
```

Use `netstat -nplt | grep tidb-server` to find out which server listens on port 4001 and run a scale-in operation for that pid:
```
tiup playground scale-in --pid 1397312
```

Now run a DDL statement:
```
create table t1(id int primary key, n int, key(n));
```

### 2. What did you expect to see? (Required)

One of these:
* DDL fails directly due to no DDL owner being available/present
* DDL completes as `tidb_enable_ddl` is ignored/reset due to the DDL owner disapearing
* DDL fails after a short while as it never starts to run
* Warnings/errors in the logs

### 3. What did you see instead (Required)

Nothing, ddl just hangs

```
mysql-8.0.11-TiDB-v8.3.0> show processlist;
+------------+------+-----------------+------+---------+------+------------+----------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------------+------+-----------------+------+---------+------+------------+----------------------------------------------------+
| 1468006408 | root | 127.0.0.1:35014 | NULL | Query | 0 | autocommit | show processlist |
| 1468006406 | root | 127.0.0.1:45618 | test | Query | 684 | autocommit | create table t1(id int primary key, n int, key(n)) |
+------------+------+-----------------+------+---------+------+------------+----------------------------------------------------+
2 rows in set (0.00 sec)
```

### 4. What is your TiDB version? (Required)

```
Release Version: v8.3.0
Edition: Community
Git Commit Hash: 1a0c3ac3292fff7742faa0c00a662ccb66ba40db
Git Branch: HEAD
UTC Build Time: 2024-08-20 10:13:01
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.