apache / apache/fluss

[Feature] Log scanner should support fetch the logs tiered into lake

Open
#41 1 comment 1 reaction 1 assignee Claimed by @luoyuxia View on GitHub
component=client
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

Currently, since lakehouse is another tiered storage of logs, log scanner should support to scan logs from the lakehouse while the remote logs has been cleaned by ttl. Also, scan log from lakehouse should be more efficient since it's columar and compressed format.

### Solution

Server can tell log scanner the logs to be fetch is in lake, and then log scanner can fetch data from lake directly. It shoudn't be much complex since the lake formats expose rich read api to help us do that.

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.