support `IMPORT INTO FROM SELECT` on DXF
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Support `IMPORT INTO ... FROM SELECT` in the NextGen execution path for materialized-view-style aggregation workloads.
The current path does not provide a worker-safe way to execute an optimized query plan across keyspaces, and HashAgg spill relies on local disk. This limits distributed import-from-query workloads when workers must use object storage.
The proposed first phase will:
- send the original SQL, source table metadata, and session execution settings to a single system-keyspace worker;
- obtain a fresh data read timestamp and build source schema from the submitted table definitions on each Query attempt; retry the complete query without retaining a task-level snapshot or GC barrier;
- load the referenced tables' persisted statistics synchronously from the target keyspace into a query-owned Domain and StatsHandle, so optimization does not depend on a warm tenant cache or async load timing;
- optimize the SELECT on the worker, discarding query/index hints while retaining TiKV/TiFlash eligibility; the plan need not be identical to the submitting TiDB's plan;
- replace worker HashAgg spill with ExternalHashAgg;
- run a dedicated Query step (query execution plus KV encode/sort), followed by the existing optional MergeSort and Ingest steps;
- fail explicitly on unsupported query capabilities, missing analyzed statistics, statistics-load errors, or resource-budget exhaustion instead of falling back to execution on the submitting TiDB.
The query-owned statistics cache must isolate tenants even when their table IDs collide. Validate cold-cache plan quality, snapshot consistency, aggregate results and S3 output. This avoids maintaining a separate physical-plan/expression serialization protocol.
Related: #60377
### Implementation PRs
Native stack #71079: #71076 → #71077 → #71078. All three heads are in `pingcap/tidb`; merge in order:
1. #71076 — Query/DXF integration using existing executors, without dedicated statistics loading.
2. #71077 — ExternalHashAgg and explicit spill configuration.
3. #71078 — Synchronous source statistics through a query-owned Domain, using the existing optimizer interfaces.
Each PR has one commit and remains Draft. GitHub manages the dependency chain as a native stack. The range-execution prototype #71068 remains a separate Draft PR outside this stack. The generic stale external-metadata overwrite risk remains tracked separately in #71063.
Contributor guide
Research direction
Start by reading the implementation stack in PRs #71076, #71077, and #71078 in order, then compare the separate range-execution prototype #71068. Check how the existing DXF Query, optional MergeSort, and Ingest steps handle the stated workload. Done means the supported import-from-query cases use the worker path with correct aggregates, snapshots, cold-cache plans, and S3 output, while unsupported cases fail explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 15/100