fetchConsumer panics (index out of range) when a topic is deleted and recreated concurrently
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
### Summary
`InMemoryStorage.fetchConsumer` can panic with `runtime error: index out of range [-1]` at [`inmemory.go:871`](https://github.com/linkedin/Burrow/blob/master/core/internal/storage/inmemory.go#L871), killing the whole Burrow process. It can also panic with an out-of-range partition index at line 863.
```
panic: runtime error: index out of range [-1]
goroutine 61 [running]:
github.com/linkedin/Burrow/core/internal/storage.(*InMemoryStorage).fetchConsumer(...)
core/internal/storage/inmemory.go:871
```
### Root cause
`fetchConsumer` snapshots the consumer group's topics under `consumerLock`, releases it, and then reads broker offsets under `brokerLock`. Storage requests are distributed across workers (`StorageSetDeleteTopic` / `StorageSetBrokerOffset` go to a random worker, `StorageSetConsumerOffset` / `StorageFetchConsumer` to a hashed worker), so a topic delete + recreate can interleave between those two phases.
Two inconsistent states are then reachable:
1. **Empty partition ring.** `deleteTopic` removes the topic from consumer groups and from the broker map under *separate* locks. A consumer commit that lands between the two halves passes the `getBrokerOffset` guard (broker state still present) and survives the delete. When the topic is recreated, `addBrokerOffset` grows the partition list with empty rings for partitions it hasn't received offsets for. `fetchConsumer` then builds an empty `partition.BrokerOffsets` slice, and since `getConsumerTopicList` always returns `Offsets` at full ring length (nils included), the `len(partition.Offsets) > 0` guard passes and `partition.BrokerOffsets[len(partition.BrokerOffsets)-1]` panics with index `[-1]`.
2. **Fewer partitions after recreate.** If the topic is recreated with fewer partitions than the consumer snapshot contains, `topicMap[p]` at line 863 indexes past the end of the broker's partition slice.
The panic is deterministic once either state exists — I reproduced it with a unit test that constructs the post-race state exactly as the interleaving leaves it (state construction in the same style as `TestInMemoryStorage_fetchConsumer_Expired`).
### Suggested fix
Mirror the existing "topic may have just been deleted" handling: skip partitions the broker no longer knows about, and only compute `CurrentLag` when at least one broker offset is present. Consumer offsets are still returned either way.
I have a PR ready with the fix plus regression tests for both states.
Burrow version: master (60d5782)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in core/internal/storage/inmemory.go at fetchConsumer around lines 863 and 871, then read TestInMemoryStorage_fetchConsumer_Expired for the existing state-construction style. Reproduce both recreated-topic states with regression tests and verify fetchConsumer returns consumer offsets without panicking while calculating lag only when broker offsets exist.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100