[STORM-3882] Continuously worker EventThread shut down when storm is running
- Dominant language
- Java
- Stars
- 6.7k
- Forks
- 4k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 62
Description
Hi everyone,
I have deployed a storm 2.4.0 cluster on my dev machine. However, when I look up the worker's log, I found that every half minute, there is a continuously zookeeper shutdown phenomenon.
While I have listened the zookeeper port, the connection number is remain the same in a long period of time, which represents that the connection between storm nimbus and zookeeper is stable.
The log record is attached as below, may I know why this zookeeper shut down phenomenon happens? And is it means something failed or it is a normal thing?
Thx!
2022-09-19 23:01:28.475 o.a.s.z.ClientZookeeper main [INFO] Starting ZK Curator
2022-09-19 23:01:28.476 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Starting
2022-09-19 23:01:28.493 o.a.s.s.o.a.z.ZooKeeper main [INFO] Initiating client connection, connectString=127.0.0.1:2182 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@528c868
2022-09-19 23:01:28.507 o.a.s.s.o.a.z.c.X509Util main [INFO] Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-09-19 23:01:28.524 o.a.s.s.o.a.z.ClientCnxnSocket main [INFO] jute.maxbuffer value is 4194304 Bytes
2022-09-19 23:01:28.539 o.a.s.s.o.a.z.ClientCnxn main [INFO] zookeeper.request.timeout value is 0. feature enabled=
2022-09-19 23:01:28.561 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Default schema
2022-09-19 23:01:28.564 o.a.s.s.o.a.z.ClientCnxn main-SendThread(127.0.0.1:2182) [INFO] Opening socket connection to server VM-24-50-centos/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error)
2022-09-19 23:01:28.575 o.a.s.s.o.a.z.ClientCnxn main-SendThread(127.0.0.1:2182) [INFO] Socket connection established, initiating session, client: /127.0.0.1:51751, server: VM-24-50-centos/127.0.0.1:2182
2022-09-19 23:01:28.585 o.a.s.s.o.a.z.ClientCnxn main-SendThread(127.0.0.1:2182) [INFO] Session establishment complete on server VM-24-50-centos/127.0.0.1:2182, sessionid = 0x10b2a6dab680068, negotiated timeout = 20000
2022-09-19 23:01:28.591 o.a.s.s.o.a.c.f.s.ConnectionStateManager main-EventThread [INFO] State change: CONNECTED
2022-09-19 23:01:28.619 o.a.s.s.o.a.c.f.i.EnsembleTracker main-EventThread [INFO] New config event received: {}
2022-09-19 23:01:28.620 o.a.s.s.o.a.c.f.i.EnsembleTracker main-EventThread [INFO] New config event received: {}
2022-09-19 23:01:28.625 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl Curator-Framework-0 [INFO] backgroundOperationsLoop exiting
2022-09-19 23:01:28.735 o.a.s.s.o.a.z.ZooKeeper main [INFO] Session: 0x10b2a6dab680068 closed
2022-09-19 23:01:28.735 o.a.s.s.o.a.z.ClientCnxn main-EventThread [INFO] EventThread shut down for session: 0x10b2a6dab680068
---
Originally reported by davinci2333, imported from: Continuously worker EventThread shut down when storm is running
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Storm 2.4.0 worker behavior and reviewing the attached Curator/ZooKeeper log sequence, especially the backgroundOperationsLoop and EventThread shutdown. Trace when the worker creates and closes the client, then determine whether the repeated shutdown is expected or indicates a failure; completion requires a confirmed explanation or a reproducible fix with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100