apache / apache/fluss

[Feature] Support batch read table without datalake enabled

Open
#40 1 comment 0 reactions 0 assignees View on GitHub
component=connector/flink
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

Currently, if a table is not with datalake enabled, Fluss won't support batch read the table. Also, although the table is datalake enabeld, it the lake snapshot doesn't exist, the batch read will also fail which will force users to wait the snapshot available.

We should resolve these two cases for better user experience

### Solution

1: In the case that the table is not with datalake enabeld, if it's primary key table, we should mrege ssts and change logs to read the full data, if it's a log table, we can just read the logs upto the latest offset.
2: In the case that the snapshot is not available for datalake enabeld, we can just consider it as a table without datalake enabeld and then batch read

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests. Start by locating the batch-read implementation and the existing primary-key, log-table, and datalake snapshot paths. Verify the design for tables without datalake enabled and for missing snapshots, including full primary-key data and log data through the latest offset.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.