pingcap / pingcap/tidb

executor: DISTRIBUTE TABLE returns an empty result set (no job id) when the distribution job finishes before the read-back

Open
#71,365 2 comments 0 reactions 1 assignee Claimed by @xzhangxian1008 View on GitHub
severity/moderate sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

Create a small partitioned table and run a distribution (observed on a pdms / PD-microservices cluster; the underlying logic is not pdms-specific):

```sql
CREATE DATABASE geopartition;
USE geopartition;
CREATE TABLE t (a INT, b INT, INDEX idx(b)) PARTITION BY RANGE(a) (
PARTITION p1 VALUES LESS THAN (10000),
PARTITION p2 VALUES LESS THAN (MAXVALUE));

DISTRIBUTE TABLE t rule='leader-scatter' engine='tikv';
```

`DISTRIBUTE TABLE` creates a distribution job, then reads the created job back from the PD scheduler config (`GET /pd/api/v1/scheduler-config/balance-range-scheduler/list`) to obtain its job id. For a small table the job finishes in ~10 ms, so the read-back can happen after the job is already `finished`.

### 2. What did you expect to see? (Required)

The id of the newly created distribution job. The statement is supposed to return it, and the unit test `pkg/executor/distribute_table_test.go` (`MockDistributePDCli{skip:1}`) asserts it is returned even when the first read fails.

### 3. What did you see instead (Required)

The statement returned an **empty result set**: no job id **and no error**, although the job had actually been created and executed.

Root cause: `pkg/executor/distribute.go` `DistributeTableExec.getSchedulerJob` matches jobs with `job["status"] != "finished"`. A job that has already `finished` is therefore skipped; after the retries (3 x 500 ms) it reports "not found", and `Next` appends no row, so the statement silently returns an empty result set.

Observed for one execution:
- The distribution job: `create=15:18:09.388324, start=15:18:09.398706, finish=15:18:09.398847, status="finished"` (finished ~10 ms after creation).
- TiDB read-back: the first GET of `/pd/api/v1/scheduler-config/balance-range-scheduler/list` returned 404 (pdms: the scheduling microservice had not yet loaded the scheduler), the following attempts got 200 but were filtered out by `status != "finished"` -> empty result set.
- `SHOW DISTRIBUTION JOBS` shows the job correctly (`status=finished`), i.e. the job exists and no data is lost.

### 4. What is your TiDB version? (Required)

`v9.0.0-beta.2.pre-2272-g6884fa5eab` (also reproduced on `v9.0.0-beta.2.pre-2270-g5316d43574`).

### Customer impact

- Functional impact: **none on the distribution itself** - the job is created and executed, and there is no data correctness, consistency or availability impact.
- API/contract impact: `DISTRIBUTE TABLE` intermittently returns **no job id** (empty result set) instead of the id. Callers can no longer reference the job (`SHOW DISTRIBUTION JOB ` / `CANCEL DISTRIBUTION JOB `), and automation / scripts / CI that parse the return value fail. Workaround: look the id up via `SHOW DISTRIBUTION JOBS`.
- Observability: the failure is **silent** (the statement succeeds with no rows), which makes it hard to detect. A caller that retries after getting no id hits `ERROR 8243 (HY000): "job already exists"`, which is confusing.
- Frequency: intermittent. More likely on pdms deployments, where the first read-back can 404 while the scheduling microservice asynchronously loads the scheduler (see tikv/pd#11285).

### Suggested fix

- Do not exclude `finished` jobs when matching the created job (match by alias/engine/rule and take the newest job id).
- If the job still cannot be read back after the retries, return an error instead of an empty result set.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.