apache / apache/fluss

[lake/tiering] Support group-aware table assignment for tiering services

Open
#3,789 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Currently, all datalake-enabled tables are scheduled through one global pending queue in `LakeTableTieringManager`. A tiering service requests the next available table through `lakeTieringHeartbeat`, and any tiering service instance may receive any table.

This works for global load balancing, but it does not provide resource isolation between different table workloads. In production, tables may have very different characteristics and requirements, for example:

- large tables and small tables need different Flink resources;
- tables may have different freshness targets or operational SLAs;
- a resource-intensive or repeatedly failing table should not affect unrelated tables;
- operators may want to upgrade or scale tiering services for one set of tables independently.

Although multiple tiering service jobs can run simultaneously, today they all consume from the same global table queue and cannot be assigned a specific set of tables.

We should introduce group-aware table assignment so that tables can be routed to tiering services in a deterministic and operationally isolated way.

### Solution

Introduce a tiering group for both tables and tiering service instances.

### Table configuration

Add a table-level option such as:

```text
table.datalake.tiering-group = default
```

The default value should be `default` for backward compatibility. The option should be dynamically configurable through `ALTER TABLE`.

### Tiering service configuration

Add a tiering service option such as:

```shell
--fluss.tiering.group default
```

A tiering service registers or reports its group in the lake tiering heartbeat. The Coordinator should only assign tables belonging to the same group.

### Coordinator scheduling

`LakeTableTieringManager` should support group-aware pending-table scheduling, for example through per-group pending queues or a group-aware `requestTable(group)` operation.

Expected behavior:

- multiple tiering services in the same group continue to share work and load balance;
- tiering services in different groups never receive each other's tables;
- existing tables and existing tiering service deployments remain in the `default` group;
- if a pending or scheduled table changes group, it moves to the new group queue;
- if a table changes group while a tiering round is running, the current round should finish under the existing epoch and subsequent rounds should use the new group.

Group information should also be exposed in relevant logs and scheduling metrics.

### Compatibility and validation

The change should be backward compatible by treating an absent group as `default`. Tests should cover:

- compatibility with existing clients and tiering services;
- isolation between different groups;
- load balancing between multiple services in the same group;
- dynamically moving a pending table to another group;
- changing the group of a table during an active tiering round;
- behavior when no tiering service is available for a group.

### Anything else?

This proposal intentionally uses explicit group assignment rather than regex-based table filtering. Regex-based selection can overlap or leave tables without an owner, while a single explicit group provides deterministic routing.

This is complementary to:

- #2360, which covers isolation of individual table failures;
- #2985, which proposes a Spark tiering service and mentions multi-table concurrency;
- #2816 and #3075, which improve visibility into lake tiering scheduling and pending tables.

A similar group-based resource isolation model is used by Apache Amoro self-optimizing, where each table is assigned to an optimizer group and optimizer instances only poll tasks from their registered group.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Read LakeTableTieringManager and the lakeTieringHeartbeat entry point first; trace table and service group configuration through ALTER TABLE and coordinator scheduling. Run the existing lake-tiering tests, then add coverage for default compatibility, same-group balancing, cross-group isolation, group changes during and between rounds, unavailable groups, and exposed group metrics or logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.