pingcap / pingcap/tidb

support `IMPORT INTO FROM SELECT` on DXF

Open
#71,043 0 comments 0 reactions 0 assignees View on GitHub
component/import nextgen type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Support `IMPORT INTO ... FROM SELECT` in the NextGen execution path for materialized-view-style aggregation workloads.

The current path does not provide a worker-safe way to execute an optimized query plan across keyspaces, and HashAgg spill relies on local disk. This limits distributed import-from-query workloads when workers must use object storage.

The proposed first phase will:

- send the original SQL, source table metadata, and session execution settings to a single system-keyspace worker;
- obtain a fresh data read timestamp and build source schema from the submitted table definitions on each Query attempt; retry the complete query without retaining a task-level snapshot or GC barrier;
- load the referenced tables' persisted statistics synchronously from the target keyspace into a query-owned Domain and StatsHandle, so optimization does not depend on a warm tenant cache or async load timing;
- optimize the SELECT on the worker, discarding query/index hints while retaining TiKV/TiFlash eligibility; the plan need not be identical to the submitting TiDB's plan;
- replace worker HashAgg spill with ExternalHashAgg;
- run a dedicated Query step (query execution plus KV encode/sort), followed by the existing optional MergeSort and Ingest steps;
- fail explicitly on unsupported query capabilities, missing analyzed statistics, statistics-load errors, or resource-budget exhaustion instead of falling back to execution on the submitting TiDB.

The query-owned statistics cache must isolate tenants even when their table IDs collide. Validate cold-cache plan quality, snapshot consistency, aggregate results and S3 output. This avoids maintaining a separate physical-plan/expression serialization protocol.

Related: #60377

### Implementation PRs

Native stack #71079: #71076 → #71077 → #71078. All three heads are in `pingcap/tidb`; merge in order:

1. #71076 — Query/DXF integration using existing executors, without dedicated statistics loading.
2. #71077 — ExternalHashAgg and explicit spill configuration.
3. #71078 — Synchronous source statistics through a query-owned Domain, using the existing optimizer interfaces.

Each PR has one commit and remains Draft. GitHub manages the dependency chain as a native stack. The range-execution prototype #71068 remains a separate Draft PR outside this stack. The generic stale external-metadata overwrite risk remains tracked separately in #71063.

Contributor guide

Open the contributing guide

Research direction

Start by reading the implementation stack in PRs #71076, #71077, and #71078 in order, then compare the separate range-execution prototype #71068. Check how the existing DXF Query, optional MergeSort, and Ingest steps handle the stated workload. Done means the supported import-from-query cases use the worker path with correct aggregates, snapshots, cold-cache plans, and S3 output, while unsupported cases fail explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.