pingcap / pingcap/tidb

Lightning replace conflict resolution fails with Error 1390: Prepared statement contains too many placeholders in v8.5.4

Open
#66,240 4 comments 0 reactions 0 assignees View on GitHub
affects-8.1 affects-8.5 component/lightning contribution severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

When importing with TiDB Lightning (conflict.strategy = "replace"), the import reaches 100% data restore but fails during post-processing with:

insert data conflict record for conflict detection 'replace' mode failed: Error 1390 (HY000): Prepared statement contains too many placeholders

This happened on TiDB v8.5.4 with ~1.88M conflict records.

Progress reached 100% (tables/chunks/engines = 100%)
Then failed in post-processing:
restore all tables data failed
run failed
the whole procedure failed
Error stack points to:
[errormanager.go] in ReplaceConflictKeys conflict-record insert path
Warning also shows:
Detect 1885595 conflict errors in total ... conflict_view

Environment

TiDB Lightning: v8.5.4
Conflict strategy: replace
Large dataset (~1.385 TiB restored, billions of rows)

In replace conflict handling, conflict rows are inserted with a single dynamically built [INSERT ... VALUES (...)]statement that can grow too large and exceed placeholder limits (65535).
Additionally, in ReplaceConflictKeys, buffered insertRows can keep accumulating across loop iterations, making the SQL even larger.

### 2. What did you expect to see? (Required)

Lightning should split conflict-record inserts into safe batches and finish post-processing without placeholder overflow.

### 3. What did you see instead (Required)
Lightning fails in post-processing with Error 1390, and the whole task reports failure even though data restore reached 100%.

### 4. What is your TiDB version? (Required)
v8.5.4

Contributor guide

Open the contributing guide

Research direction

Start in errormanager.go at ReplaceConflictKeys and trace the conflict-record insert path used by TiDB Lightning's replace strategy. Check how insertRows is buffered across loop iterations and how the dynamically built INSERT statement is formed. Done means conflict records are inserted in safe batches and post-processing completes without Error 1390.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.