apache / apache/doris

[Bug](High) Flexible Partial Update fails with ROW binlog due to illegal partial update block columns

Open
#65,866 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Tracking

- Tracking issue: #65265

### Search before asking

- [x] I searched the existing issues and found no issue with the same error or reproduction.

### Version

Reproduced while testing #62606 on Doris commit `a78b707ed040c3206df99cf43ea3101ab2ba5b90`.

Test environment: non-Cloud deployment with 1 FE and 3 BEs.

### What's Wrong?

Flexible Partial Update fails when it is used on a UNIQUE KEY merge-on-write table with ROW binlog enabled. The Stream Load reaches the BE flush stage and returns:

```text
[INTERNAL_ERROR] illegal partial update block columns: 10, num key columns: 1, total schema columns: 12
```

The entire batch fails with `NumberLoadedRows=0`.

The same schema accepts Flexible Partial Update when ROW binlog is disabled, and fixed-column partial update works with ROW binlog enabled. This narrows the failure to the combination of ROW binlog and `UPDATE_FLEXIBLE_COLUMNS`.

IVM requires ROW binlog, so the issue blocks IVM base tables from using Flexible Partial Update.

### What You Expected?

Flexible Partial Update should write successfully while ROW binlog records sufficient before/after data for downstream incremental consumption. A later incremental refresh should produce results consistent with the base table.

### How to Reproduce?

Create the table and initial row:

```sql
DROP DATABASE IF EXISTS ivm_flexible_row_binlog_repro;
CREATE DATABASE ivm_flexible_row_binlog_repro;
USE ivm_flexible_row_binlog_repro;

CREATE TABLE flexible_base (
id BIGINT NOT NULL,
seq BIGINT NOT NULL,
amount INT NULL,
status VARCHAR(32) NULL,
comment VARCHAR(64) NULL
)
UNIQUE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
'replication_num'='1',
'enable_unique_key_merge_on_write'='true',
'enable_unique_key_skip_bitmap_column'='true',
'function_column.sequence_col'='seq',
'binlog.enable'='true',
'binlog.format'='ROW',
'binlog.need_historical_value'='true'
);

INSERT INTO flexible_base VALUES (1, 10, 100, 'new', 'one');
```

Prepare `flexible.json`:

```json
{"id":1,"seq":11,"amount":111}
{"id":2,"seq":1,"status":"inserted"}
```

Run Flexible Partial Update:

```bash
curl -u root: -H 'format:json' -H 'read_json_by_line:true' \
-H 'unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS' \
-H 'Expect:100-continue' \
-T flexible.json \
http://FE_HOST:8030/api/ivm_flexible_row_binlog_repro/flexible_base/_stream_load
```

The Stream Load fails during BE flush with the error above and writes zero rows.

### Regression Test Result

- Suite: `ivm_dml_flexible_partial_row_binlog_blocking`
- Jenkins: `regression-debug #793`
- Result: 4 DML suites were run; the other 3 passed and only this suite failed with the INTERNAL_ERROR.

### Anything Else?

The failure occurs below the IVM refresh layer in the storage flush path, so changing the IVM refresh strategy cannot avoid it.

### Are you willing to submit PR?

- [ ] Yes, I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by running the ivm_dml_flexible_partial_row_binlog_blocking regression suite and reproduce the Stream Load failure with UPDATE_FLEXIBLE_COLUMNS and ROW binlog enabled. Trace the BE flush path described in the issue; done means the batch writes successfully, ROW binlog retains sufficient before/after data, and the incremental refresh remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.