apache / apache/fluss

Streaming union read from the lake and Fluss log table using a specified offset timestamp

Open
#3,230 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

I found that we cannot consume the log table using hybrid splits (lake + log) when a specific timestamp is provided.
My intention is to perform a streaming read starting from a given timestamp, but part of the data may reside in the underlying lakehouse rather than the log layer.

Image

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named in the issue. Start by tracing how hybrid splits choose lake and log data when a timestamp is supplied, then identify the streaming read entry point and relevant tests. Done means a read from the specified timestamp can combine data from both the underlying lakehouse and the Fluss log table.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.