linkedin / linkedin/Burrow

Burrow doesn't detect kafka consumers that are added after Burrow has started up

Open
#487 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

When I add a kafka consumer after burrow is up and running, burrow doesn't see the consumer unless I restart burrow.

**Steps to reproduce:**

1. Clone burrow, check out v1.2.0
2. Edit docker-config/burrow.toml to remove the group-blacklist line for console consumers in the `[consumer.local]` section
3. Run `docker-compose up` from the checkout root
4. Call `curl http://0.0.0.0:8000/v3/kafka/local/topic` and verify that the expected topics have been created (after a minute or so):
`{"error":false,"message":"topic list returned","topics":["test-topic","__consumer_offsets","test-topic2","test-topic3"],"request":{"url":"/v3/kafka/local/topic","host":"cb2d49f194dd"}}`
5. Call `curl http://0.0.0.0:8000/v3/kafka/local/consumer` and verify that there are currently no consumers (as expected)
6. Create a console producer and consumer, and send a few messages.
7. Run `curl http://0.0.0.0:8000/v3/kafka/local/consumer` and observe that there still no consumers (though we just created one, and sent it some messages)
8. Wait for awhile. Still no consumer found when hitting `http://0.0.0.0:8000/v3/kafka/local/consumer`
9. Restart burrow: `docker restart burrow_burrow_1`. Consumer shows up immediately after burrow restarts.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue using docker-config/burrow.toml and docker-compose up, then query the /v3/kafka/local/consumer endpoint before and after creating a console consumer. Trace how consumer discovery is refreshed after startup. Done means the newly created consumer appears without restarting Burrow.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, go, kafka
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.