linkedin / linkedin/Burrow

fetchConsumer panics (index out of range) when a topic is deleted and recreated concurrently

Open
#864 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

### Summary

`InMemoryStorage.fetchConsumer` can panic with `runtime error: index out of range [-1]` at [`inmemory.go:871`](https://github.com/linkedin/Burrow/blob/master/core/internal/storage/inmemory.go#L871), killing the whole Burrow process. It can also panic with an out-of-range partition index at line 863.

```
panic: runtime error: index out of range [-1]

goroutine 61 [running]:
github.com/linkedin/Burrow/core/internal/storage.(*InMemoryStorage).fetchConsumer(...)
core/internal/storage/inmemory.go:871
```

### Root cause

`fetchConsumer` snapshots the consumer group's topics under `consumerLock`, releases it, and then reads broker offsets under `brokerLock`. Storage requests are distributed across workers (`StorageSetDeleteTopic` / `StorageSetBrokerOffset` go to a random worker, `StorageSetConsumerOffset` / `StorageFetchConsumer` to a hashed worker), so a topic delete + recreate can interleave between those two phases.

Two inconsistent states are then reachable:

1. **Empty partition ring.** `deleteTopic` removes the topic from consumer groups and from the broker map under *separate* locks. A consumer commit that lands between the two halves passes the `getBrokerOffset` guard (broker state still present) and survives the delete. When the topic is recreated, `addBrokerOffset` grows the partition list with empty rings for partitions it hasn't received offsets for. `fetchConsumer` then builds an empty `partition.BrokerOffsets` slice, and since `getConsumerTopicList` always returns `Offsets` at full ring length (nils included), the `len(partition.Offsets) > 0` guard passes and `partition.BrokerOffsets[len(partition.BrokerOffsets)-1]` panics with index `[-1]`.

2. **Fewer partitions after recreate.** If the topic is recreated with fewer partitions than the consumer snapshot contains, `topicMap[p]` at line 863 indexes past the end of the broker's partition slice.

The panic is deterministic once either state exists — I reproduced it with a unit test that constructs the post-race state exactly as the interleaving leaves it (state construction in the same style as `TestInMemoryStorage_fetchConsumer_Expired`).

### Suggested fix

Mirror the existing "topic may have just been deleted" handling: skip partitions the broker no longer knows about, and only compute `CurrentLag` when at least one broker offset is present. Consumer offsets are still returned either way.

I have a PR ready with the fix plus regression tests for both states.

Burrow version: master (60d5782)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in core/internal/storage/inmemory.go at fetchConsumer around lines 863 and 871, then read TestInMemoryStorage_fetchConsumer_Expired for the existing state-construction style. Reproduce both recreated-topic states with regression tests and verify fetchConsumer returns consumer offsets without panicking while calculating lag only when broker offsets exist.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
backend, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.