More graceful ErrOffsetOutOfRange handling
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
**Go version:** 1.14.2
**Burrow version:** github.com/linkedin/Burrow v1.3.4-0.20200506150011-4ce194fea01a
We have lately been seeing (seemingly random) `ErrOffsetOutOfRange` errors for two specific partitions of the `__consumer_offsets` topic and the internal Burrow consumer. This has started happening after upgrading our cluster to Kafka version 2.4.
We have not found the reason for this, and it may be related to our cluster setup. But the error, `ErrOffsetOutOfRange`, causes a nil-pointer in Burrow after the error is logged.
```
{"level":"ERROR","@timestamp":"2020-05-27T08:14:04.090Z","caller":"runtime/asm_amd64.s:1373","message":"consume error","@version":"1","type":"module","coordinator":"consumer","class":"kafka","name":"prod","topic":"__consumer_offsets","partition":9,"error":"kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.","stacktrace":"github.com/linkedin/Burrow/core/internal/consumer.(*KafkaClient).partitionConsumer\n\t/go/pkg/mod/github.com/linkedin/!burrow@v1.3.4-0.20200506150011-4ce194fea01a/core/internal/consumer/kafka_client.go:261"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x9ca109]
goroutine 393 [running]:
github.com/linkedin/Burrow/core/internal/consumer.(*KafkaClient).partitionConsumer(0xc00042edc0, 0xdf2b40, 0xc000ddb4a0, 0x0)
/go/pkg/mod/github.com/linkedin/!burrow@v1.3.4-0.20200506150011-4ce194fea01a/core/internal/consumer/kafka_client.go:262 +0x5e9
created by github.com/linkedin/Burrow/core/internal/consumer.(*KafkaClient).startKafkaConsumer
/go/pkg/mod/github.com/linkedin/!burrow@v1.3.4-0.20200506150011-4ce194fea01a/core/internal/consumer/kafka_client.go:314 +0x968
```
The underlying Sarama partition consumer considers the error fatal, and require user action:
https://github.com/Shopify/sarama/blob/b5764af1c47d0f6718dba3be6a3d75e8c97b351a/consumer.go#L826-L828
I would be nice if Burrow could handle the error more gracefully, and possibly restart the partition consumer at `OffsetNewest`.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read core/internal/consumer/kafka_client.go, especially partitionConsumer around line 261 and startKafkaConsumer around line 314, then compare the Sarama partition-consumer behavior linked in the issue. Confirm the ErrOffsetOutOfRange path does not cause a nil-pointer panic and determine whether the partition consumer can restart at OffsetNewest.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100