google / google/symphony-gcp

Operator readiness probe fails during high usage

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

### Description

During high usage, the operator readiness probe crashes.

Image

In the image above the yellow lines represent readiness probe failure.
The readiness probe failed while returning 5k pods.
The CPU limits and requests where set to 4 vCPU. The memory was set to 8Gi.

Bellow is the corresponding traceback

```
2025-09-04 12:46:24.830 Exception occurred during processing of request from ('10.1.201.1', 50464)
2025-09-04 12:46:24.830 Traceback (most recent call last):
2025-09-04 12:46:24.830 File "/usr/local/lib/python3.13/socketserver.py", line 318, in _handle_request_noblock
2025-09-04 12:46:24.830 self.process_request(request, client_address)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 349, in process_request
2025-09-04 12:46:24.831 self.finish_request(request, client_address)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 362, in finish_request
2025-09-04 12:46:24.831 self.RequestHandlerClass(request, client_address, self)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/site-packages/gcp_symphony_operator/workers/health_server.py", line 65, in
2025-09-04 12:46:24.831 ("", self.port), lambda *args: HealthHandler(self.config, *args)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/site-packages/gcp_symphony_operator/workers/health_server.py", line 13, in __init__
2025-09-04 12:46:24.831 super().__init__(*args)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 766, in __init__
2025-09-04 12:46:24.831 self.handle()
2025-09-04 12:46:24.831 ~~~~~~~~~~~^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/http/server.py", line 436, in handle
2025-09-04 12:46:24.831 self.handle_one_request()
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~~~~~~^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/http/server.py", line 424, in handle_one_request
2025-09-04 12:46:24.831 method()
2025-09-04 12:46:24.831 ~~~~~~^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/site-packages/gcp_symphony_operator/workers/health_server.py", line 30, in do_GET
2025-09-04 12:46:24.831 self.wfile.write(b"Ready")
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~^^^^^^^^^^
2025-09-04 12:46:24.831 File "/usr/local/lib/python3.13/socketserver.py", line 845, in write
2025-09-04 12:46:24.831 self._sock.sendall(b)
2025-09-04 12:46:24.831 ~~~~~~~~~~~~~~~~~~^^^
2025-09-04 12:46:24.831 BrokenPipeError: [Errno 32] Broken pipe
```

### Version

Repository hash: 700ff4f8ae089919221985a95484416527febbd4
Operator version: 0.2.4

Contributor guide

Open the contributing guide

Research direction

Start with workers/health_server.py, especially HealthHandler.do_GET, and reproduce the readiness probe under the reported high-usage conditions with roughly 5k pods. Use the traceback and the reported operator version and repository hash to investigate the failure; done means the readiness probe no longer crashes or fails during that workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.