Non-transactional DML: make the shard column check smarter
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
### Setup:
```sql
CREATE TABLE t (
id bigint PRIMARY KEY AUTO_INCREMENT,
user_id bigint NOT NULL,
state tinyint NOT NULL,
create_at int NOT NULL,
KEY idx_01 (user_id, state, create_at)
);
INSERT INTO t (user_id, state, create_at) VALUES
(1,2,100),(1,2,101),(1,2,102),(1,2,103),(1,2,104),(2,1,100)......;
```
### Background
BATCH splits one DML into many small ones. Given:
```sql
BATCH ON id LIMIT 2
UPDATE t SET state=3 WHERE user_id=1 AND state=2 AND create_at <= 200;
```
TiDB reads the matching id values in sorted order (1..5), cuts them into groups of at most LIMIT rows, and runs one ordinary statement per group (confirmed by DRY RUN):
```sql
UPDATE t SET state=3 WHERE id BETWEEN 1 AND 2 AND (user_id=1 AND state=2 AND create_at <= 200);
UPDATE t SET state=3 WHERE id BETWEEN 3 AND 4 AND (...);
UPDATE t SET state=3 WHERE id BETWEEN 5 AND 5 AND (...);
```
Each small statement must locate its rows through an index on the shard column (id BETWEEN ... above); otherwise every batch scans the whole table, and N batches scan it N times. To rule that out, the shard column must be the first column of some index
```go
func selectShardColumnByGivenName(shardColumnName string, tbl table.Table) (
......
for _, index := range tbl.Indices() {
if index.Meta().State != model.StatePublic || index.Meta().Invisible {
continue
}
indexColumns := index.Meta().Columns
// check only the first column
if len(indexColumns) > 0 && indexColumns[0].Name.L == shardColumnName {
indexed = true
break
}
}
return indexed, shardColumnInfo, nil
}
```
This rule is cheap but errs in both directions:
| # | Statement | Check | Actual behavior | Verdict |
|---|-----------|-------|-----------------|---------|
| 1 | `BATCH ON create_at LIMIT 2 ... WHERE create_at <= 200` | rejected | each batch could only `TableFullScan` → N× amplification | correct rejection |
| 2 | `BATCH ON create_at LIMIT 2 ... WHERE user_id=1 AND state=2 AND create_at <= 200` | rejected | both phases can use `idx_01` (plans below) | **false positive** |
| 3 | `BATCH ON user_id LIMIT 2 ... WHERE user_id=1 AND state=2 AND create_at <= 200` | accepted | degenerates to a single batch of 5 rows (`user_id BETWEEN 1 AND 1`), exceeding LIMIT 2 | **false negative** |
| 4 | `BATCH ON id LIMIT 2 ...` (any WHERE) | accepted | PK: always range-scannable, unique, total ≈ one pass | correct acceptance |
Case 2 evidence — the equality conditions pin the prefix of idx_01:
```sql
-- batch-dividing query: IndexRangeScan on idx_01, keep order:true (no Sort)
EXPLAIN SELECT create_at FROM t
WHERE user_id=1 AND state=2 AND create_at <= 200 ORDER BY create_at;
-- per-batch statement: IndexRangeScan on idx_01, range:[1 2 100, 1 2 101]
EXPLAIN UPDATE t SET state=3
WHERE create_at BETWEEN 100 AND 101 AND (user_id=1 AND state=2 AND create_at <= 200);
```
Case 3 evidence — DRY RUN shows a single split statement:
```sql
UPDATE `t` SET `state`=3 WHERE (`user_id` BETWEEN 1 AND 1 AND (...))
```
### Proposals
1. Accept the k-th column of an index when the WHERE clause has top-level AND-ed constant equality conditions on columns 1..k-1 of that index; anything more complex (OR, expressions, non-constants) keeps the current rejection. — fixes case 2
2. Warn (or error) at validation time when the WHERE clause pins the shard column itself to a constant: a single batch is then guaranteed. — fixes case 3, no data knowledge needed
Contributor guide
Research direction
Start at selectShardColumnByGivenName and use the supplied EXPLAIN and DRY RUN examples as behavioral cases. Trace how index prefixes and shard-column equality are validated, then cover cases 2 and 3: accept a usable indexed prefix and warn or error when the shard column is pinned to a constant. Done means case 1 remains rejected and the proposed false positives and false negatives are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100