[STORM-167] proposal for storm topology online update

https://github.com/nathanmarz/storm/issues/540

Now update topology code can only be done by kill it and re-submit a new one. During the kill and re-submit process some request may delay or fail. It is not so good for online service. So we consider to add topology online update recently.

Mission

update running topology code gracefully one worker after another without service total interrupted. Just update topology code, not update topology DAG structure including component, stream and task number.

Proposal

• client use "storm update topology-name new-jar-file" to submit new-jar-file update request


• nimbus update stormdist dir, link topology-dir to new one


• nimbus update topology version on zk


• the supervisors that running this topology update it
  
  
  
  • check topology version on zk, if it is not the same as local version, a topology update begin
  
  
  • each supervisor schedule the topology's worker update at a rand(expect-max-update-time) time point
  
  
  • sync-supervisor download the latest code from nimbus
  
  
  • sync-process check local worker heartbeat version(to be added), if it is not the same with sync-supervisor downloaded version, kill the worker
  
  
  • sync-process restart killed worker
  
  
  • new worker heartbeat to zk with version(to be added), it can be displayed on web ui to check update progress.
  
  
  
  

This feature is deployed in our production clusters. It's really useful for topologys handling online request waiting for response. Topology jar can be updated without entire service offline.

We hope that this feature is useful for others too.

---
Originally reported by xumingming, imported from: proposal for storm topology online update

• assignee: parth.brahmbhatt
  
• status: Closed
  
• priority: Minor
  
• resolution: Abandoned
  
• resolved: 2024-07-02T10:54:46+01:00
  
• imported: 2025-01-24
  

The issue names the client update command, Nimbus, ZooKeeper versioning, supervisors, sync-supervisor, and sync-process; start by tracing those entry points and read linked issue #540. Done means a running topology can receive a new jar with workers updated one at a time, without changing the DAG or fully interrupting service, with progress visible through worker heartbeat versions.

Written by the indexing model from the issue text.