astronomy-commons / astronomy-commons/adc-streaming

Problem handling Kafka exceptions

Open
#54 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
6
PR merge metrics
No merged PRs in 30d

Description

I am still unsure of the underlying cause and whether this problem is endemic to all installations but it is repeatable on my development workstation, an Intel iMac at home.

The symptom is a slew of errors like:
```
%5|1647498302.003|REQTMOUT|rdkafka#consumer-2| [thrd:sasl_ssl://kb-2.prod.hop.scimma.org:9092/2]: sasl_ssl://kb-2.prod.hop.scimma.org:9092/2: Timed out FetchRequest in flight (after 1799295ms, timeout #0)

%4|1647498302.005|REQTMOUT|rdkafka#consumer-2| [thrd:sasl_ssl://kb-0.prod.hop.scimma.org:9092/0]: sasl_ssl://kb-0.prod.hop.scimma.org:9092/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1647498302.005|FAIL|rdkafka#consumer-2| [thrd:sasl_ssl://kb-0.prod.hop.scimma.org:9092/0]: sasl_ssl://kb-0.prod.hop.scimma.org:9092/0: 1 request(s) timed out: disconnect (after 115074987ms in state UP)
```
The Exception is raised in: /Users/areeda/mambaforge/envs/igwn-py39/lib/python3.9/site-packages/adc/errors.py line 12:

My problem is that this kills the main thread of our DQR alert listener without any way to catch the exception.

I have a merge request going in next.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading adc/errors.py at line 12 and trace how the Kafka exception reaches the DQR alert listener's main thread. Use the reported REQTMOUT and FAIL messages to investigate reproduction on the stated development setup, then review the merge request mentioned in the issue. Done means the exception-handling behavior is defined and the listener no longer terminates unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, python
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.