linkedin / linkedin/Burrow

More graceful ErrOffsetOutOfRange handling

Open
#634 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

**Go version:** 1.14.2
**Burrow version:** github.com/linkedin/Burrow v1.3.4-0.20200506150011-4ce194fea01a

We have lately been seeing (seemingly random) `ErrOffsetOutOfRange` errors for two specific partitions of the `__consumer_offsets` topic and the internal Burrow consumer. This has started happening after upgrading our cluster to Kafka version 2.4.

We have not found the reason for this, and it may be related to our cluster setup. But the error, `ErrOffsetOutOfRange`, causes a nil-pointer in Burrow after the error is logged.

```
{"level":"ERROR","@timestamp":"2020-05-27T08:14:04.090Z","caller":"runtime/asm_amd64.s:1373","message":"consume error","@version":"1","type":"module","coordinator":"consumer","class":"kafka","name":"prod","topic":"__consumer_offsets","partition":9,"error":"kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.","stacktrace":"github.com/linkedin/Burrow/core/internal/consumer.(*KafkaClient).partitionConsumer\n\t/go/pkg/mod/github.com/linkedin/!burrow@v1.3.4-0.20200506150011-4ce194fea01a/core/internal/consumer/kafka_client.go:261"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x9ca109]

goroutine 393 [running]:
github.com/linkedin/Burrow/core/internal/consumer.(*KafkaClient).partitionConsumer(0xc00042edc0, 0xdf2b40, 0xc000ddb4a0, 0x0)
/go/pkg/mod/github.com/linkedin/!burrow@v1.3.4-0.20200506150011-4ce194fea01a/core/internal/consumer/kafka_client.go:262 +0x5e9
created by github.com/linkedin/Burrow/core/internal/consumer.(*KafkaClient).startKafkaConsumer
/go/pkg/mod/github.com/linkedin/!burrow@v1.3.4-0.20200506150011-4ce194fea01a/core/internal/consumer/kafka_client.go:314 +0x968
```

The underlying Sarama partition consumer considers the error fatal, and require user action:
https://github.com/Shopify/sarama/blob/b5764af1c47d0f6718dba3be6a3d75e8c97b351a/consumer.go#L826-L828

I would be nice if Burrow could handle the error more gracefully, and possibly restart the partition consumer at `OffsetNewest`.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read core/internal/consumer/kafka_client.go, especially partitionConsumer around line 261 and startKafkaConsumer around line 314, then compare the Sarama partition-consumer behavior linked in the issue. Confirm the ErrOffsetOutOfRange path does not cause a nil-pointer panic and determine whether the partition consumer can restart at OffsetNewest.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.