[STORM-3798] Nimbus keep on crash once restart nimbus machine

I have setup the storm cluster in AWS environment with 6 node clusters. 1 Master and 5 slave and the zookeeper are installed in 3 node clusters.

 

When I shutdown zookeeper and storm nodes every day night and start in the morning, When the machine boots up I have started all the nimbus and supervisor and zookeeper service via supervisord. Once machine up all the service are running but in the nimbus logs I am getting "java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?" error.

 

I have restarted zookeeper and nimbus also not working still getting this error. Once I remove all the data from zookeeper data directly then restart zookeeper and storm service then working fine.

Why I have to do this everyday when the machine starts up? how can I fix this issue. When machine boot up all the service should work fine.

---
Originally reported by anandh.kumar, imported from: Nimbus keep on crash once restart nimbus machine

• status: Open
  
• priority: Major
  
• resolution: Unresolved
  
• imported: 2025-01-24
  

Start with the Nimbus logs and the supervisord startup configuration, comparing the service startup order with the three-node ZooKeeper setup. Reproduce the shutdown and reboot cycle without deleting ZooKeeper data, and use the "No nimbus leader participant host found" error to identify the failure. Done means Nimbus and the supervisors recover after reboot without clearing ZooKeeper data.

Written by the indexing model from the issue text.