apache / apache/fluss

[Feature] Add fine-grained table synchronization for lakehouse compaction service.

Open
#186 4 comments 0 reactions 1 assignee Claimed by @leonardBang View on GitHub
component=lake
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

The lakehouse compaction service is coarse-grained now, in some scene, I'd like to appoint specific tables to synchronous rather than all tables in database. In addition, when there are many tables in database, synchronous all tables will cause performance issues, fine-grained table synchronization is more realistic.

### Solution

1. Add new regular expressions for tables like appointed database now.
2. In class `FlinkSourceEnumerator`, filter mismatched tables or partitions.

### Anything else?

_No response_

### Willingness to contribute

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.