Connector lag monitoring support
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Hi,
I'm currently giving a try with Burrow and running into some problems.
I'm trying to monitor lag for connectors (HDFS, JDBC, etc) and when I'm starting burrow on my test environment I get the following debug trace :
```
{"level":"debug","ts":1554735418.3128974,"msg":"skipped metadata because of unknown protocolType","type":"module","coordinator":"consumer","class":"kafka","name":"consumer_kafka","offset_topic":"__consumer_offsets","offset_partition":3,"offset_offset":1117,"message_type":"metadata","group":"connector.dev._connect_topic-namexxxx-mz0zx7t8u7-cluster","protocol_type":"connect","generation":1,"protocol":"default","leader":"connect-1-d4fcb6c5-91af-4f6a-a735-ad82fd748c36","current_state_timestamp":1553520383214}
{"level":"debug","ts":1554735418.3129158,"msg":"skipped metadata because of unknown protocolType","type":"module","coordinator":"consumer","class":"kafka","name":"consumer_kafka","offset_topic":"__consumer_offsets","offset_partition":3,"offset_offset":1118,"message_type":"metadata","group":"connector.dev._connect_topic-namexx-sourcexxxx-mz0zx7t8u7-cluster","protocol_type":"connect","generation":2,"protocol":"default","leader":"connect-1-d4fcb6c5-91af-4f6a-a735-ad82fd748c36","current_state_timestamp":1553520384339}
```
So I guess `connect` is not a supported protocolType ?
I see that this code was added 3 months ago and apparently is the cause of this `pruning`.
https://github.com/linkedin/Burrow/blob/3cdc7e73b8388985d95ea7f05ef23c7bec3e9985/core/internal/consumer/kafka_client.go#L433-L437
Is there any plan to add this support in the future ? or this is more a PEBKAC ?
FYI : I also tested to remove this code and I got another error when extracting the group metadata. It fail with a `"failed to decode"` `reason:assigment` here :
https://github.com/linkedin/Burrow/blob/3cdc7e73b8388985d95ea7f05ef23c7bec3e9985/core/internal/consumer/kafka_client.go#L588-L591
And caused by an error in the member assignment extraction : https://github.com/linkedin/Burrow/blob/3cdc7e73b8388985d95ea7f05ef23c7bec3e9985/core/internal/consumer/kafka_client.go#L611-L614
But I'm not sure that those two errors are related.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in core/internal/consumer/kafka_client.go, especially the protocolType handling around lines 433-437 and group metadata decoding around lines 588-614. Reproduce the connector-group trace, then determine whether the connect protocol can be supported without the reported metadata and member-assignment decoding failures. Done means connector lag groups are handled without being pruned or failing to decode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100