apache / apache/paimon

[Bug] Scan from-snapshot-full and compact-full won't produce a snapshot in streaming mode with LogStore when start up

Open
#2,236 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.5-release

### Compute Engine

flink

### Minimal reproduce step

1. set log store
2. set scan.mode = 'from-snapshot-full' or 'compacted-full'
3. set streaming-read-mode = log
4. streaming scan the table

### What doesn't meet your expectations?

In FlinkSourceBuilder#build(), there won't produce from the start snapshot on the table upon first startup, it will read the log directly. Which is unexpected.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in FlinkSourceBuilder#build() and reproduce the listed setup with a LogStore, streaming-read-mode set to log, and scan.mode set to from-snapshot-full or compacted-full. Trace the startup scan path and verify that an initial table snapshot is produced before log records are read in both modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.