[Bug](High) Flexible Partial Update fails with ROW binlog due to illegal partial update block columns
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Tracking
- Tracking issue: #65265
### Search before asking
- [x] I searched the existing issues and found no issue with the same error or reproduction.
### Version
Reproduced while testing #62606 on Doris commit `a78b707ed040c3206df99cf43ea3101ab2ba5b90`.
Test environment: non-Cloud deployment with 1 FE and 3 BEs.
### What's Wrong?
Flexible Partial Update fails when it is used on a UNIQUE KEY merge-on-write table with ROW binlog enabled. The Stream Load reaches the BE flush stage and returns:
```text
[INTERNAL_ERROR] illegal partial update block columns: 10, num key columns: 1, total schema columns: 12
```
The entire batch fails with `NumberLoadedRows=0`.
The same schema accepts Flexible Partial Update when ROW binlog is disabled, and fixed-column partial update works with ROW binlog enabled. This narrows the failure to the combination of ROW binlog and `UPDATE_FLEXIBLE_COLUMNS`.
IVM requires ROW binlog, so the issue blocks IVM base tables from using Flexible Partial Update.
### What You Expected?
Flexible Partial Update should write successfully while ROW binlog records sufficient before/after data for downstream incremental consumption. A later incremental refresh should produce results consistent with the base table.
### How to Reproduce?
Create the table and initial row:
```sql
DROP DATABASE IF EXISTS ivm_flexible_row_binlog_repro;
CREATE DATABASE ivm_flexible_row_binlog_repro;
USE ivm_flexible_row_binlog_repro;
CREATE TABLE flexible_base (
id BIGINT NOT NULL,
seq BIGINT NOT NULL,
amount INT NULL,
status VARCHAR(32) NULL,
comment VARCHAR(64) NULL
)
UNIQUE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
'replication_num'='1',
'enable_unique_key_merge_on_write'='true',
'enable_unique_key_skip_bitmap_column'='true',
'function_column.sequence_col'='seq',
'binlog.enable'='true',
'binlog.format'='ROW',
'binlog.need_historical_value'='true'
);
INSERT INTO flexible_base VALUES (1, 10, 100, 'new', 'one');
```
Prepare `flexible.json`:
```json
{"id":1,"seq":11,"amount":111}
{"id":2,"seq":1,"status":"inserted"}
```
Run Flexible Partial Update:
```bash
curl -u root: -H 'format:json' -H 'read_json_by_line:true' \
-H 'unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS' \
-H 'Expect:100-continue' \
-T flexible.json \
http://FE_HOST:8030/api/ivm_flexible_row_binlog_repro/flexible_base/_stream_load
```
The Stream Load fails during BE flush with the error above and writes zero rows.
### Regression Test Result
- Suite: `ivm_dml_flexible_partial_row_binlog_blocking`
- Jenkins: `regression-debug #793`
- Result: 4 DML suites were run; the other 3 passed and only this suite failed with the INTERNAL_ERROR.
### Anything Else?
The failure occurs below the IVM refresh layer in the storage flush path, so changing the IVM refresh strategy cannot avoid it.
### Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
Contributor guide
Research direction
Start by running the ivm_dml_flexible_partial_row_binlog_blocking regression suite and reproduce the Stream Load failure with UPDATE_FLEXIBLE_COLUMNS and ROW binlog enabled. Trace the BE flush path described in the issue; done means the batch writes successfully, ROW binlog retains sufficient before/after data, and the incremental refresh remains consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100