[server] Adjust ZooKeeper session timeout to prevent ISR shrink before leader failover
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Currently, Fluss's default configuration for ZooKeeper session timeout and follower replica lag time can lead to availability issues during network failures. The parameter causing ISR (In-Sync Replicas) to shrink before the leader is detected as dead, potentially leaving no eligible leader candidates.
**Current Configuration in Fluss:**
- `zookeeper.client.session-timeout`: **60 seconds** (defined in `ConfigOptions.ZOOKEEPER_SESSION_TIMEOUT`)
- `log.replica.max-lag-time`: **30 seconds** (defined in `ConfigOptions.LOG_REPLICA_MAX_LAG_TIME`)
- Actual ISR shrink timing: approximately 45 seconds (1.5x max lag time)
**Problem Scenario:**
When a leader (TabletServer A) experiences network issues:
1. **At ~30-45 seconds**: Followers (B, C) cannot fetch data from A and are removed from ISR due to `log.replica.max-lag-time` timeout
2. **At ~60 seconds**: ZooKeeper session expires, and A is marked as dead
3. **Result**: When leader election is triggered, ISR only contains A (already dead), and B/C are not eligible candidates → **No available leader, partition becomes unavailable**
**Kafka's Configuration (Reference):**
From [Kafka 2.5+](https://kafka.apache.org/26/getting-started/upgrade/#notable-changes-in-250):
- `zookeeper.session.timeout.ms`: **18 seconds** (increased from 6s for better resiliency)
- `replica.lag.time.max.ms`: **30 seconds** (increased from 10s)
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating ConfigOptions.ZOOKEEPER_SESSION_TIMEOUT and ConfigOptions.LOG_REPLICA_MAX_LAG_TIME, then trace the ISR shrink and leader-failover entry points described in the issue. Reproduce or test the network-failure timing, and consider the work complete when followers remain eligible until the failed leader is detected and the partition elects an available leader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100