apache / apache/kyuubi

[Bug] Socket Timeout When Connecting to Kyuubi Server

Open
#7,006 4 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

When attempting to connect to the Kyuubi server, the connection intermittently fails due to a socket timeout issue. Logs indicate that the connection to the ZooKeeper server is lost and then re-established. However, this results in disruptions when attempting to interact with Kyuubi, causing instability in client connections.

### Affects Version(s)

master

### Kyuubi Server Log Output

```logtalk
[
{
"timeMillis": 1743312078141,
"timestamp": "2025-03-30T05:21:18.141+0000",
"thread": "main-SendThread(zookeeper.nprod-green-dev-zookeeper.svc.cluster.local:2181)",
"threadId": 20,
"threadPriority": 5,
"level": "INFO",
"loggerName": "org.apache.kyuubi.shaded.zookeeper.ClientCnxn",
"message": "Unable to read additional data from server sessionid 0x2007639831c0000, likely server has closed socket, closing socket connection and attempting reconnect",
"contextMap": {},
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
},
{
"timeMillis": 1743312080239,
"timestamp": "2025-03-30T05:21:20.239+0000",
"thread": "main-SendThread(zookeeper.nprod-green-dev-zookeeper.svc.cluster.local:2181)",
"threadId": 20,
"threadPriority": 5,
"level": "INFO",
"loggerName": "org.apache.kyuubi.shaded.zookeeper.ClientCnxn",
"message": "Opening socket connection to server zookeeper.nprod-green-dev-zookeeper.svc.cluster.local/10.97.33.246:2181. Will not attempt to authenticate using SASL (unknown error)",
"contextMap": {},
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
},
{
"timeMillis": 1743312080240,
"timestamp": "2025-03-30T05:21:20.240+0000",
"thread": "main-SendThread(zookeeper.nprod-green-dev-zookeeper.svc.cluster.local:2181)",
"threadId": 20,
"threadPriority": 5,
"level": "INFO",
"loggerName": "org.apache.kyuubi.shaded.zookeeper.ClientCnxn",
"message": "Socket connection established to zookeeper.nprod-green-dev-zookeeper.svc.cluster.local/10.97.33.246:2181, initiating session",
"contextMap": {},
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
},
{
"timeMillis": 1743312080252,
"timestamp": "2025-03-30T05:21:20.252+0000",
"thread": "main-EventThread",
"threadId": 21,
"threadPriority": 5,
"level": "INFO",
"loggerName": "org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager",
"message": "State change: RECONNECTED",
"contextMap": {},
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
},
{
"timeMillis": 1743312080253,
"timestamp": "2025-03-30T05:21:20.253+0000",
"thread": "Curator-ConnectionStateManager-0",
"threadId": 19,
"threadPriority": 5,
"level": "INFO",
"loggerName": "org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient",
"message": "Zookeeper client connection state changed to: RECONNECTED",
"contextMap": {},
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
},
{
"timeMillis": 1743312080327,
"timestamp": "2025-03-30T05:21:20.327+0000",
"thread": "main-EventThread",
"threadId": 21,
"threadPriority": 5,
"level": "WARN",
"loggerName": "org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient",
"message": "This Kyuubi instance 0.0.0.0:10009 now receives the NodeDataChanged event",
"contextMap": {},
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
}
]
```

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [x] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by investigating the reported ZooKeeper and Curator connection-state sequence around the socket timeout and reconnect, then reproduce the intermittent client disruption. No source file, test, configuration, or entry point is identified in the issue; done would require a confirmed cause and a verified fix with regression coverage.

Written by the indexing model from the issue text.

Assessment

Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.