influxdata / influxdata/telegraf

Cancel gracefully when connecting to kafka broker

Open
#12,214 1 comment 0 reactions 0 assignees View on GitHub
area/kafka help wanted size/m
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

If the user sends a signal to telegraf while kafka_consumer is connecting to the broker (before sarama.NewConsumerGroup() has returned), the signal will be delayed until connection is made or timed out. This means sigint to shut down telegraf, or sighup to reload config are ignored while connecting to the broker.

Since #12111, telegraf allows the kafka connection to be deferred until after startup and to retry for a very long time in case the broker is down. This means the time spent connecting to the broker can be much larger and signals can be blocked indefinitely.

It doesn't look like there is a way to cancel sarama.NewConsumerGroup(). We need to find out if there is a different way to connect that has a way to cancel, or work with upstream to add one.

Contributor guide

Open the contributing guide

Research direction

Start with the kafka_consumer connection path around sarama.NewConsumerGroup() and review whether Sarama provides a cancellable connection method. Determine how deferred startup and retry behavior interact with SIGINT and SIGHUP handling. Done means connection attempts can be interrupted so shutdown and configuration reload signals are not blocked indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.