Operator readiness probe fails during high usage
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
### Description
During high usage, the operator readiness probe crashes.
In the image above the yellow lines represent readiness probe failure.
The readiness probe failed while returning 5k pods.
The CPU limits and requests where set to 4 vCPU. The memory was set to 8Gi.
Bellow is the corresponding traceback
```
2025-09-04 12:46:24.830 Exception occurred during processing of request from ('10.1.201.1', 50464)
2025-09-04 12:46:24.830 Traceback (most recent call last):
2025-09-04 12:46:24.830 File "/usr/local/lib/python3.13/socketserver.py", line 318, in _handle_request_noblock
2025-09-04 12:46:24.830 self.process_request(request, client_address)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 349, in process_request
2025-09-04 12:46:24.831 self.finish_request(request, client_address)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 362, in finish_request
2025-09-04 12:46:24.831 self.RequestHandlerClass(request, client_address, self)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/site-packages/gcp_symphony_operator/workers/health_server.py", line 65, in
2025-09-04 12:46:24.831 ("", self.port), lambda *args: HealthHandler(self.config, *args)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/site-packages/gcp_symphony_operator/workers/health_server.py", line 13, in __init__
2025-09-04 12:46:24.831 super().__init__(*args)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 766, in __init__
2025-09-04 12:46:24.831 self.handle()
2025-09-04 12:46:24.831 ~~~~~~~~~~~^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/http/server.py", line 436, in handle
2025-09-04 12:46:24.831 self.handle_one_request()
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~~~~~^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/http/server.py", line 424, in handle_one_request
2025-09-04 12:46:24.831 method()
2025-09-04 12:46:24.831 ~~~~~~^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/site-packages/gcp_symphony_operator/workers/health_server.py", line 30, in do_GET
2025-09-04 12:46:24.831 self.wfile.write(b"Ready")
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 845, in write
2025-09-04 12:46:24.831 self._sock.sendall(b)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~^^^
2025-09-04 12:46:24.831 BrokenPipeError: [Errno 32] Broken pipe
```
### Version
Repository hash: 700ff4f8ae089919221985a95484416527febbd4
Operator version: 0.2.4
Contributor guide
Research direction
Start with workers/health_server.py, especially HealthHandler.do_GET, and reproduce the readiness probe under the reported high-usage conditions with roughly 5k pods. Use the traceback and the reported operator version and repository hash to investigate the failure; done means the readiness probe no longer crashes or fails during that workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100