executor: DISTRIBUTE TABLE returns an empty result set (no job id) when the distribution job finishes before the read-back
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
Create a small partitioned table and run a distribution (observed on a pdms / PD-microservices cluster; the underlying logic is not pdms-specific):
```sql
CREATE DATABASE geopartition;
USE geopartition;
CREATE TABLE t (a INT, b INT, INDEX idx(b)) PARTITION BY RANGE(a) (
PARTITION p1 VALUES LESS THAN (10000),
PARTITION p2 VALUES LESS THAN (MAXVALUE));
DISTRIBUTE TABLE t rule='leader-scatter' engine='tikv';
```
`DISTRIBUTE TABLE` creates a distribution job, then reads the created job back from the PD scheduler config (`GET /pd/api/v1/scheduler-config/balance-range-scheduler/list`) to obtain its job id. For a small table the job finishes in ~10 ms, so the read-back can happen after the job is already `finished`.
### 2. What did you expect to see? (Required)
The id of the newly created distribution job. The statement is supposed to return it, and the unit test `pkg/executor/distribute_table_test.go` (`MockDistributePDCli{skip:1}`) asserts it is returned even when the first read fails.
### 3. What did you see instead (Required)
The statement returned an **empty result set**: no job id **and no error**, although the job had actually been created and executed.
Root cause: `pkg/executor/distribute.go` `DistributeTableExec.getSchedulerJob` matches jobs with `job["status"] != "finished"`. A job that has already `finished` is therefore skipped; after the retries (3 x 500 ms) it reports "not found", and `Next` appends no row, so the statement silently returns an empty result set.
Observed for one execution:
- The distribution job: `create=15:18:09.388324, start=15:18:09.398706, finish=15:18:09.398847, status="finished"` (finished ~10 ms after creation).
- TiDB read-back: the first GET of `/pd/api/v1/scheduler-config/balance-range-scheduler/list` returned 404 (pdms: the scheduling microservice had not yet loaded the scheduler), the following attempts got 200 but were filtered out by `status != "finished"` -> empty result set.
- `SHOW DISTRIBUTION JOBS` shows the job correctly (`status=finished`), i.e. the job exists and no data is lost.
### 4. What is your TiDB version? (Required)
`v9.0.0-beta.2.pre-2272-g6884fa5eab` (also reproduced on `v9.0.0-beta.2.pre-2270-g5316d43574`).
### Customer impact
- Functional impact: **none on the distribution itself** - the job is created and executed, and there is no data correctness, consistency or availability impact.
- API/contract impact: `DISTRIBUTE TABLE` intermittently returns **no job id** (empty result set) instead of the id. Callers can no longer reference the job (`SHOW DISTRIBUTION JOB ` / `CANCEL DISTRIBUTION JOB `), and automation / scripts / CI that parse the return value fail. Workaround: look the id up via `SHOW DISTRIBUTION JOBS`.
- Observability: the failure is **silent** (the statement succeeds with no rows), which makes it hard to detect. A caller that retries after getting no id hits `ERROR 8243 (HY000): "job already exists"`, which is confusing.
- Frequency: intermittent. More likely on pdms deployments, where the first read-back can 404 while the scheduling microservice asynchronously loads the scheduler (see tikv/pd#11285).
### Suggested fix
- Do not exclude `finished` jobs when matching the created job (match by alias/engine/rule and take the newest job id).
- If the job still cannot be read back after the retries, return an error instead of an empty result set.
Contributor guide
Assessment
This issue has not been assessed yet.