pingcap / pingcap/tidb

support distributed single-table export to object storage (EXPORT TABLE) in nextgen

Open
#68,984 2 comments 0 reactions 0 assignees View on GitHub
nextgen type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

Exporting a large table currently relies on Dumpling, which runs on a single process outside the cluster, cannot scale with the cluster, and is hard to operate on nextgen deployments where compute is auto-scaled and the only durable storage is object storage.

**Describe the feature you'd like:**

A SQL statement `EXPORT TABLE db.t TO 's3://...' FORMAT 'csv' WITH thread=..., file_size=..., detached` that runs as a DXF task:

- the scheduler splits the table into key-ordered, region-aligned spans; each subtask runs a read -> encode -> upload pipeline (cop scan at a fixed TSO snapshot, CSV encoding via the shared text-protocol serializer, multipart upload to object storage)
- output files are ordered by handle, non-overlapping, and named so lexicographic order equals key order, compatible with IMPORT INTO round-trip
- a service safepoint protects the snapshot for the lifetime of the task
- on nextgen: submitted from the user keyspace, executed by DXF service nodes in the SYSTEM keyspace

**Describe alternatives you've considered:**

Dumpling (external process, no DXF integration, no auto-scaling); SELECT INTO OUTFILE (server-local file only).

**Teachability, Documentation, Adoption, Migration Strategy:**

Statement and job management modeled on IMPORT INTO (`SHOW EXPORT JOB`, `CANCEL EXPORT JOB` planned). Design doc and implementation plan are tracked in-repo under docs/plans/.

## Sub-PR breakdown (task list)

The feature is landed as small, independently-reviewable PRs, grouped by category. Checked = merged.

**Foundation (shared refactor)**
- [x] format, server: extract text-protocol value serializer into `pkg/format/textrow` — #69087
- [x] dumpformat/csvfile: shared `CSVWriter` — #69187
- [x] dumpformat/sqlfile: shared `SQLWriter` — #69460

**System table**
- [ ] meta, session: add `mysql.tidb_export_jobs` system table (table creation only, usage deferred) — #69089

**Scheduler / autoscaling interface**
- [ ] dxf: expose coprocessor-worker required count in `/dxf/schedule/status` (placeholder `0` first; old `tidb_worker`/`tikv_worker` semantics unchanged, for gradual control-plane migration) — #69114

**Statement & job management**
- [ ] parser: `EXPORT TABLE` / `SHOW EXPORT JOB[S]` / `CANCEL EXPORT JOB`
- [ ] executor: plan node + builder wiring + statement/option validation (stub; DXF submission deferred, parser not wired) — #69291
- [ ] executor: submit / validate / poll + `SHOW` / `CANCEL` (job-table accessors)

**DXF export task**
- [ ] dxf/export: register `Export` task type + key-ordered span split (reuse backfill region split) — #70153
- [ ] dxf/export: subtask read -> encode -> write pipeline (streaming multipart upload) + GC barrier — #70165
- [ ] dxf/export: `PostProcess` (schema/metadata files) + synchronous revert cleanup + progress aggregation

**Output formats**
- [ ] parquet encoder

**Metrics & E2E**
- [ ] metrics + Grafana panels
- [ ] realtikv e2e: IMPORT INTO round-trip + Dumpling byte-identity

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.