astronomy-commons / astronomy-commons/adc-streaming
Problem handling Kafka exceptions
- Dominant language
- Python
- Stars
- 6
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
I am still unsure of the underlying cause and whether this problem is endemic to all installations but it is repeatable on my development workstation, an Intel iMac at home.
The symptom is a slew of errors like:
```
%5|1647498302.003|REQTMOUT|rdkafka#consumer-2| [thrd:sasl_ssl://kb-2.prod.hop.scimma.org:9092/2]: sasl_ssl://kb-2.prod.hop.scimma.org:9092/2: Timed out FetchRequest in flight (after 1799295ms, timeout #0)
%4|1647498302.005|REQTMOUT|rdkafka#consumer-2| [thrd:sasl_ssl://kb-0.prod.hop.scimma.org:9092/0]: sasl_ssl://kb-0.prod.hop.scimma.org:9092/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1647498302.005|FAIL|rdkafka#consumer-2| [thrd:sasl_ssl://kb-0.prod.hop.scimma.org:9092/0]: sasl_ssl://kb-0.prod.hop.scimma.org:9092/0: 1 request(s) timed out: disconnect (after 115074987ms in state UP)
```
The Exception is raised in: /Users/areeda/mambaforge/envs/igwn-py39/lib/python3.9/site-packages/adc/errors.py line 12:
My problem is that this kills the main thread of our DQR alert listener without any way to catch the exception.
I have a merge request going in next.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading adc/errors.py at line 12 and trace how the Kafka exception reaches the DQR alert listener's main thread. Use the reported REQTMOUT and FAIL messages to investigate reproduction on the stated development setup, then review the merge request mentioned in the issue. Done means the exception-handling behavior is defined and the listener no longer terminates unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, python
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100