google / google/symphony-gcp

Operator throttles on large scaledown

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

### Issue Description

On a **real test scenario**, i.e. using real compute images and correctly sized pod resources, the operator takes long time to scale down, see image bellow:

Image

Notably, the last delete event happened after 2221 seconds (~37 minutes) after the delete request.

Compare against how it scales up:

Image

This test was ran using 3 CPUs and 3 Gb per pod (Symphony compute node) and using a total of 5000 pods. Host Factory settings were set to the default (notably the request and return batching config set to 100).

During this test the operator reached a max CPU usage of about ~1.2 and a max memory usage of ~9.6 Gb.
The operator presented a spike in memory usage at the time of the request deletion.

### Version

Repository hash: f510c68a43122bacf3f1ef6015ce40323227713c

Contributor guide

Open the contributing guide

Research direction

Reproduce the 5,000-pod scaledown with 3 CPUs and 3 GB per pod, using the default Host Factory request and return batch size of 100. Trace the operator's scale-down path around the deletion request and compare it with scale-up; done means the delay and memory spike are resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.