pingcap / pingcap/tidb

global sort: stale subtask executors may overwrite committed external metadata

Open
#71,063 2 comments 0 reactions 0 assignees View on GitHub
component/DXF component/global-sort component/import severity/minor type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This report is based on static analysis of master. The following fault-injection scenario has **not been executed**; no production incident or wrong result is claimed.

1. Run a distributed global-sort IMPORT INTO with at least two eligible TiDB workers. Record the external result metadata produced by worker A for one subtask.
2. Hold A's result-metadata object-store write in flight and make the scheduler reassign that subtask to worker B. The injection must allow an already-issued write to finish late; sleeping before a call that immediately observes a canceled context is not sufficient.
3. Let B finish the same subtask, write its result metadata to `//meta.json`, and commit the successful DXF subtask row. Record B's object contents and file list.
4. Release A's delayed write to that same object key after B's write has completed.
5. Compare the successful subtask row and the object read by downstream planning. Verify whether A can replace the result metadata associated with B's committed completion even though A no longer owns the subtask.

### 2. What did you expect to see? (Required)

After a subtask completion is committed, downstream readers should use external metadata belonging to that completion. A stale executor should not be able to change the published result by overwriting its object key.

Investigate this invariant across DXF tasks that publish external metadata under shared mutable keys. Framework-level task ownership checks and external artifact publication need to agree.

### 3. What did you see instead (Required)

The inspected code permits overlapping executions during reassignment and uses a fixed result-metadata key for global-sort IMPORT steps:

- [`newChunkWorker`](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/dxf/importinto/encode_and_sort_operator.go#L120-L125) explicitly accounts for two nodes running the same subtask after a network partition and gives their data writers distinct UUIDs.
- Encode, merge-sort and ingest completion write result metadata to a deterministic subtask path ([encode](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/dxf/importinto/task_executor.go#L423-L427), [merge](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/dxf/importinto/task_executor.go#L573-L577), [ingest](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/dxf/importinto/task_executor.go#L816-L819)). [`WriteJSONToExternalStorage`](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/ingestor/globalsort/util.go#L253-L263) writes that key without an executor/generation ownership condition.
- The object write happens inside RunSubtask, before [`finishSubtask`](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/dxf/framework/taskexecutor/task_executor.go#L525-L539). [`FinishSubtask`](https://github.com/pingcap/tidb/blob/e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e/pkg/dxf/framework/storage/subtask_state.go#L53-L60) checks `exec_id` when updating the database row, but that check does not protect the earlier object-store write.

The suspected outcome is a committed row referring to a metadata object subsequently replaced by a stale execution. The replacement can be a complete valid object; this does not depend on exposing a partially uploaded file.

User-visible impact still needs validation. For deterministic retries, the two file lists may describe equivalent data. Other cases may expose mismatched artifacts or task failure, but this report does **not** establish data corruption or a reproduced wrong-result case. Tracking severity: moderate.

Related reports were checked: #61537 concerns cross-cluster URI collisions, #69802 concerns orphaned conflicted-row objects, and #70408 concerns persisted plans after resource reduction. They do not cover this delayed same-subtask metadata overwrite scenario.

### 4. What is your TiDB version? (Required)

Audited upstream/master commit `e26d86f9fb58c1829e75ac7f8bf6caa91bd4693e`. Release-version impact has not been established.

No runtime `SELECT tidb_version()` output is available because the proposed fault-injection reproduction has not been run.

Contributor guide

Open the contributing guide

Research direction

Start with newChunkWorker in pkg/dxf/importinto/encode_and_sort_operator.go, the encode, merge, and ingest completion paths in task_executor.go, and WriteJSONToExternalStorage in pkg/ingestor/globalsort/util.go. Trace the write through RunSubtask and FinishSubtask, then validate the delayed-write reassignment scenario; done means a stale executor cannot replace metadata after another execution commits the subtask.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.