Burrow doesn't detect kafka consumers that are added after Burrow has started up
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
When I add a kafka consumer after burrow is up and running, burrow doesn't see the consumer unless I restart burrow.
**Steps to reproduce:**
1. Clone burrow, check out v1.2.0
2. Edit docker-config/burrow.toml to remove the group-blacklist line for console consumers in the `[consumer.local]` section
3. Run `docker-compose up` from the checkout root
4. Call `curl http://0.0.0.0:8000/v3/kafka/local/topic` and verify that the expected topics have been created (after a minute or so):
`{"error":false,"message":"topic list returned","topics":["test-topic","__consumer_offsets","test-topic2","test-topic3"],"request":{"url":"/v3/kafka/local/topic","host":"cb2d49f194dd"}}`
5. Call `curl http://0.0.0.0:8000/v3/kafka/local/consumer` and verify that there are currently no consumers (as expected)
6. Create a console producer and consumer, and send a few messages.
7. Run `curl http://0.0.0.0:8000/v3/kafka/local/consumer` and observe that there still no consumers (though we just created one, and sent it some messages)
8. Wait for awhile. Still no consumer found when hitting `http://0.0.0.0:8000/v3/kafka/local/consumer`
9. Restart burrow: `docker restart burrow_burrow_1`. Consumer shows up immediately after burrow restarts.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue using docker-config/burrow.toml and docker-compose up, then query the /v3/kafka/local/consumer endpoint before and after creating a console consumer. Trace how consumer discovery is refreshed after startup. Done means the newly created consumer appears without restarting Burrow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, go, kafka
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100