support distributed single-table export to object storage (EXPORT TABLE) in nextgen
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
Exporting a large table currently relies on Dumpling, which runs on a single process outside the cluster, cannot scale with the cluster, and is hard to operate on nextgen deployments where compute is auto-scaled and the only durable storage is object storage.
**Describe the feature you'd like:**
A SQL statement `EXPORT TABLE db.t TO 's3://...' FORMAT 'csv' WITH thread=..., file_size=..., detached` that runs as a DXF task:
- the scheduler splits the table into key-ordered, region-aligned spans; each subtask runs a read -> encode -> upload pipeline (cop scan at a fixed TSO snapshot, CSV encoding via the shared text-protocol serializer, multipart upload to object storage)
- output files are ordered by handle, non-overlapping, and named so lexicographic order equals key order, compatible with IMPORT INTO round-trip
- a service safepoint protects the snapshot for the lifetime of the task
- on nextgen: submitted from the user keyspace, executed by DXF service nodes in the SYSTEM keyspace
**Describe alternatives you've considered:**
Dumpling (external process, no DXF integration, no auto-scaling); SELECT INTO OUTFILE (server-local file only).
**Teachability, Documentation, Adoption, Migration Strategy:**
Statement and job management modeled on IMPORT INTO (`SHOW EXPORT JOB`, `CANCEL EXPORT JOB` planned). Design doc and implementation plan are tracked in-repo under docs/plans/.
## Sub-PR breakdown (task list)
The feature is landed as small, independently-reviewable PRs, grouped by category. Checked = merged.
**Foundation (shared refactor)**
- [x] format, server: extract text-protocol value serializer into `pkg/format/textrow` — #69087
- [x] dumpformat/csvfile: shared `CSVWriter` — #69187
- [x] dumpformat/sqlfile: shared `SQLWriter` — #69460
**System table**
- [ ] meta, session: add `mysql.tidb_export_jobs` system table (table creation only, usage deferred) — #69089
**Scheduler / autoscaling interface**
- [ ] dxf: expose coprocessor-worker required count in `/dxf/schedule/status` (placeholder `0` first; old `tidb_worker`/`tikv_worker` semantics unchanged, for gradual control-plane migration) — #69114
**Statement & job management**
- [ ] parser: `EXPORT TABLE` / `SHOW EXPORT JOB[S]` / `CANCEL EXPORT JOB`
- [ ] executor: plan node + builder wiring + statement/option validation (stub; DXF submission deferred, parser not wired) — #69291
- [ ] executor: submit / validate / poll + `SHOW` / `CANCEL` (job-table accessors)
**DXF export task**
- [ ] dxf/export: register `Export` task type + key-ordered span split (reuse backfill region split) — #70153
- [ ] dxf/export: subtask read -> encode -> write pipeline (streaming multipart upload) + GC barrier — #70165
- [ ] dxf/export: `PostProcess` (schema/metadata files) + synchronous revert cleanup + progress aggregation
**Output formats**
- [ ] parquet encoder
**Metrics & E2E**
- [ ] metrics + Grafana panels
- [ ] realtikv e2e: IMPORT INTO round-trip + Dumpling byte-identity
Contributor guide
Assessment
This issue has not been assessed yet.