apache / apache/fluss

[server] Adjust ZooKeeper session timeout to prevent ISR shrink before leader failover

Open
#2,708 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Currently, Fluss's default configuration for ZooKeeper session timeout and follower replica lag time can lead to availability issues during network failures. The parameter causing ISR (In-Sync Replicas) to shrink before the leader is detected as dead, potentially leaving no eligible leader candidates.

**Current Configuration in Fluss:**

- `zookeeper.client.session-timeout`: **60 seconds** (defined in `ConfigOptions.ZOOKEEPER_SESSION_TIMEOUT`)
- `log.replica.max-lag-time`: **30 seconds** (defined in `ConfigOptions.LOG_REPLICA_MAX_LAG_TIME`)
- Actual ISR shrink timing: approximately 45 seconds (1.5x max lag time)

**Problem Scenario:**

When a leader (TabletServer A) experiences network issues:

1. **At ~30-45 seconds**: Followers (B, C) cannot fetch data from A and are removed from ISR due to `log.replica.max-lag-time` timeout
2. **At ~60 seconds**: ZooKeeper session expires, and A is marked as dead
3. **Result**: When leader election is triggered, ISR only contains A (already dead), and B/C are not eligible candidates → **No available leader, partition becomes unavailable**

**Kafka's Configuration (Reference):**

From [Kafka 2.5+](https://kafka.apache.org/26/getting-started/upgrade/#notable-changes-in-250):
- `zookeeper.session.timeout.ms`: **18 seconds** (increased from 6s for better resiliency)
- `replica.lag.time.max.ms`: **30 seconds** (increased from 10s)

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating ConfigOptions.ZOOKEEPER_SESSION_TIMEOUT and ConfigOptions.LOG_REPLICA_MAX_LAG_TIME, then trace the ISR shrink and leader-failover entry points described in the issue. Reproduce or test the network-failure timing, and consider the work complete when followers remain eligible until the failed leader is detected and the partition elects an available leader.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.