pingcap / pingcap/tidb

lightning: add target-partition config to ingest directly into a named partition

Open
#67,894 0 comments 1 reaction 0 assignees View on GitHub
component/tablepartition contribution type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Summary

TiDB Lightning currently writes all SSTs into the full table key range when loading into a partitioned table. This makes it impossible to run concurrent Lightning jobs targeting different partitions of the same table — they conflict during the local sort and ingest phase.

## Motivation

For customers with large partitioned tables (e.g. LIST COLUMNS partitioned by customer), we need to run many independent Lightning jobs in parallel, each loading data for one partition. Today this requires a temp table + EXCHANGE PARTITION dance which is slow and error-prone.

## Proposed Change

Add a `target-partition` config option to `[mydumper]`:

```toml
[mydumper]
target-partition = "p_acme_2026_04_19"
```

When set, Lightning scopes all operations to the named partition:
- **Empty pre-check**: checks only the target partition
- **Checksum**: runs `ADMIN CHECKSUM TABLE t PARTITION (p)` instead of full-table
- **ANALYZE**: runs `ANALYZE TABLE t PARTITION (p)` instead of full-table

This also requires extending the `ADMIN CHECKSUM TABLE` grammar to accept a `PARTITION (...)` clause.

## Related PR

https://github.com/pingcap/tidb/pull/67892

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.