Operator throttles on large scaledown
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
### Issue Description
On a **real test scenario**, i.e. using real compute images and correctly sized pod resources, the operator takes long time to scale down, see image bellow:
Notably, the last delete event happened after 2221 seconds (~37 minutes) after the delete request.
Compare against how it scales up:
This test was ran using 3 CPUs and 3 Gb per pod (Symphony compute node) and using a total of 5000 pods. Host Factory settings were set to the default (notably the request and return batching config set to 100).
During this test the operator reached a max CPU usage of about ~1.2 and a max memory usage of ~9.6 Gb.
The operator presented a spike in memory usage at the time of the request deletion.
### Version
Repository hash: f510c68a43122bacf3f1ef6015ce40323227713c
Contributor guide
Research direction
Reproduce the 5,000-pod scaledown with 3 CPUs and 3 GB per pod, using the default Host Factory request and return batch size of 100. Trace the operator's scale-down path around the deletion request and compare it with scale-up; done means the delay and memory spike are resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100