Delete consumer groups across multiple burrow instances

Nobody has claimed this yet.

Currently it is possible to run multiple instances of burrow, monitoring the same kafka cluster. Only one instance will alert because of the notifier lock on zookeeper, which is great.

In the event that a a group needs to be deleted, it needs to be deleted from all running instances and not just the one that has the notifier lock. If Burrow's rest api is exposed over a load balancer, it can be a little tricky to make sure that all running instances have the group removed. Wondering what some ideas might be so all running instances can communicate to each other, so if one instance gets a delete, it can communicate that to the others.

The simple answer is probably share the deletes on a kafka topic, but wasn't sure if we wanted to require kafka topics to be able to run burrow. Want to start a discussion on best approach before I go off implementing something.

@toddpalino curious how you guys are currently handling this. Do you mainly run one instance, or do you not have to delete groups very often?

Start with Burrow's REST API delete operation and the notifier lock on ZooKeeper, then compare the issue's proposed Kafka-topic coordination with other possible approaches. The issue is currently a design discussion, so done requires an agreed cross-instance deletion behavior and acceptance criteria before implementation can begin.

Written by the indexing model from the issue text.