Question: How to observe consumer lag, in a centralized way, in confluent cloud

Nobody has claimed this yet.

Hi,

We want to centralize the way we handle consumer lag. Currently we have a number of services deployed in an aws ECS fargate cluster. Each service is configured to receive a statics payload which we inspect for a consumer lag and then publish a custom metric into cloudwatch, which allows us to act accordingly. The downside to this approach is that we have to update any new services with this code. Also, its reliant on the driver providing this feature and not all our services are in the written in the same language.

Burrow would appear to be the tool for the job. Although, it seems from my reading its doesn't work in confluent cloud. Is this still the case?

If Burrow cannot work in confluent cloud we see two other options described below. Your thoughts would be greatly appreciated:

**Option 1.**

Create a new ECS service in our cluster which ran a script which would in turn run the "kafka-consumer-group" console admin tool on an interval and then publish the cloudwatch metric. In your experience does these seem like a sensible approach that could work?

**Option 2.**

The other I've seen, that may work in confluent cloud is to setup an alert on the consumer lag in control-center and then have a lambda in aws issue a rest-api request to Get the alert history on a schedule and then push the metric from there

Start by reviewing Burrow's support for Confluent Cloud and its Kafka consumer-group monitoring path; the issue names no file or test. Determine whether Burrow can observe consumer lag in Confluent Cloud and document a supported centralized approach, including how it compares with the proposed ECS and Control Center options.

Written by the indexing model from the issue text.