containers / containers/nri-plugins
balloons: Synchronize() may not finish within timeout with 100 containers and 600+ IRQs
- Dominant language
- Go
- Stars
- 114
- Forks
- 44
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 20
Description
### Bug Description
Starting balloons pod may fail due to exceeding container runtime's plugin_request_timeout that defaults to 2 seconds. The reason is that synchronization may take very long on a large server that has already 100+ containers running. Noticed when enabling IRQ controls in such a system.
### Expected behavior
Synchronization should handle 100 containers within the default timeout.
### Experienced behavior
It did not.
### Steps to reproduce the behavior
Deploy the balloons plugin with IRQ configuration enabled on a server with 600+ IRQs to be managed and 100 containers running. Optionally include CPU class configurations that configure frequencies with cpufreq or SST.
### Environment
- nri-plugins version: e8e29bff9619bdc6cbb075a2ab18620017cbe514 (v0.13.1 + balloons IRQ support)
- nri plugin(s) running: balloons
- container runtime: containerd
### Additional context
Perhaps Synchronize should not apply system configurations (cpufreq, IRQ, ...) for each container separately, as long as there are containers left that may immediately make previous configuration invalid at least in some parts.
Contributor guide
Research direction
Start with the balloons plugin's Synchronize() path and reproduce the issue using IRQ configuration with 600+ IRQs and about 100 containers, optionally including cpufreq or SST settings. Trace which synchronization work consumes the plugin_request_timeout and consider the issue complete when synchronization handles that workload within the default two-second timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100