apache / apache/paimon

[Bug] Sequence group may cause record in wrong state.

Open
#5,387 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

If my records come disorderly, -D record may have been compact out of table. Then +I (which sequence number is smaller then -D record) may survive.

StreamWriteBuilder builder = getTable().newStreamWriteBuilder();
StreamTableWrite write = builder.newWrite();
write.withIOManager(new IOManagerImpl("~/temp/"));
StreamTableCommit commit = builder.newCommit();

// +I[2025-04-01, 1, type1, 29, 5, name2, 29, 5, name2, null, 9223372036854775802]
GenericRow genericRow = GenericRow.of(
1L,
2L,
2L);

GenericRow genericRow2 = GenericRow.ofKind(RowKind.DELETE,
1L,
null,
Long.MAX_VALUE - 1);

GenericRow genericRow3 = GenericRow.of(
1L,
1L,
1L);

write.write(genericRow);
commit.commit(1L, write.prepareCommit(true, 1L));

write.write(genericRow2);
commit.commit(2L, write.prepareCommit(true, 2L));
readData();

write.write(genericRow3);
commit.commit(3L, write.prepareCommit(true, 3L));

Record should be 1,null,Long.MAX_VALUE-1

### Compute Engine

Flink

### Minimal reproduce step

above

### What doesn't meet your expectations?

--

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided minimal reproduction and trace the sequence-group behavior through StreamWriteBuilder, StreamTableWrite, StreamTableCommit, and readData. Verify the disorderly +I and -D sequence described in the issue, then confirm that the final record is 1,null,Long.MAX_VALUE-1.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.