Zookeeper Transaction

There are places in the code base in which multiple write interactions with ZK is done as a part of one operation. If any of these interactions fails or if the pinot component fails between these ZK interactions, then we'll be in an inconsistent state. For example we have this situation in segment commit end when a consuming segment gets committed. To clean up the issues from the mentioned failures, we have set up a periodic task (segment validation manager job), and periodically look for these inconsistencies and try to fix them.

A better approach is to use the [ZK Transaction API](https://zookeeper.apache.org/doc/r3.4.13/api/org/apache/zookeeper/Transaction.html) to prevent having these inconsistencies in the first place. At the beginning of the operation, we can create a ZK transaction object and then use the transaction object to interact with ZK by:
- creating new ZNode
- modifying an existing ZNode
- deleting an existing ZNode

When ZK operations are done, then we commit all of them at once. If commit is successful, then all ZK operations have successfully completed, otherwise none will be applied.

By briefly looking at Helix API's, it looks like Helix doesn't expose ZK transaction API's. Until Helix provides the transaction API's, I think we should directly use Zookeeper client to leverage transaction capabilities which, in turns, reduces the chances of facing the mentioned failure cases. It'll also help simplifying the code base to handle these edge cases which is getting more complicated by adding new features.

Start at the segment commit end operation and inspect the segment validation manager job to understand the inconsistencies it currently repairs. Review the Helix APIs and the linked Zookeeper Transaction API before deciding how the relevant ZK interactions could be grouped. Done means the multi-step operation is applied atomically, with no partial ZK state when a step or the Pinot component fails.

Written by the indexing model from the issue text.