chime-experiment / chime-experiment/coco

metrics endpoint crashes

Open
#204 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3
Forks
2
PR merge metrics
No merged PRs in 30d

Description

We created an endpoint for testing that just waits 6minutes:
```
group: cluster
call:
coco:
name: wait
request:
duration: 360s
```

This was to test if it makes open redis connections time out, but we found that the metrics endpoint also crashes (if it get's called during the wait-time, probably):
```
May 29 12:24:50 csBfs cocod[27856]: ----------------------------------------
May 29 12:24:50 csBfs cocod[27856]: Exception happened during processing of request from ('10.1.111.8', 56538)
May 29 12:24:50 csBfs cocod[27856]: Traceback (most recent call last):
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 650, in process_request_thread
May 29 12:24:50 csBfs cocod[27856]: self.finish_request(request, client_address)
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request
May 29 12:24:50 csBfs cocod[27856]: self.RequestHandlerClass(request, client_address, self)
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__
May 29 12:24:50 csBfs cocod[27856]: self.handle()
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/http/server.py", line 426, in handle
May 29 12:24:50 csBfs cocod[27856]: self.handle_one_request()
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request
May 29 12:24:50 csBfs cocod[27856]: method()
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/site-packages/coco/metric.py", line 43, in do_GET
May 29 12:24:50 csBfs cocod[27856]: self.wfile.write(output)
May 29 12:24:50 csBfs cocod[27856]: File "/usr/local/lib/python3.7/socketserver.py", line 799, in write
May 29 12:24:50 csBfs cocod[27856]: self._sock.sendall(b)
May 29 12:24:50 csBfs cocod[27856]: BrokenPipeError: [Errno 32] Broken pipe
May 29 12:24:50 csBfs cocod[27856]: ----------------------------------------
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with coco/metric.py at line 43 and inspect how the metrics handler writes its response. Reproduce the six-minute wait while requesting the metrics endpoint, then verify that a disconnected client no longer produces the reported BrokenPipeError or crashes the service.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.