chime-experiment / chime-experiment/coco
metrics endpoint crashes
- Dominant language
- C++
- Stars
- 3
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
We created an endpoint for testing that just waits 6minutes:
```
group: cluster
call:
coco:
name: wait
request:
duration: 360s
```
This was to test if it makes open redis connections time out, but we found that the metrics endpoint also crashes (if it get's called during the wait-time, probably):
```
May 29 12:24:50 csBfs cocod[27856]: ----------------------------------------
May 29 12:24:50 csBfs cocod[27856]: Exception happened during processing of request from ('10.1.111.8', 56538)
May 29 12:24:50 csBfs cocod[27856]: Traceback (most recent call last):
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 650, in process_request_thread
May 29 12:24:50 csBfs cocod[27856]: self.finish_request(request, client_address)
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request
May 29 12:24:50 csBfs cocod[27856]: self.RequestHandlerClass(request, client_address, self)
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__
May 29 12:24:50 csBfs cocod[27856]: self.handle()
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/http/server.py", line 426, in handle
May 29 12:24:50 csBfs cocod[27856]: self.handle_one_request()
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request
May 29 12:24:50 csBfs cocod[27856]: method()
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/site-packages/coco/metric.py", line 43, in do_GET
May 29 12:24:50 csBfs cocod[27856]: self.wfile.write(output)
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 799, in write
May 29 12:24:50 csBfs cocod[27856]: self._sock.sendall(b)
May 29 12:24:50 csBfs cocod[27856]: BrokenPipeError: [Errno 32] Broken pipe
May 29 12:24:50 csBfs cocod[27856]: ----------------------------------------
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with coco/metric.py at line 43 and inspect how the metrics handler writes its response. Reproduce the six-minute wait while requesting the metrics endpoint, then verify that a disconnected client no longer produces the reported BrokenPipeError or crashes the service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100