containers / containers/nri-plugins

balloons: Synchronize() may not finish within timeout with 100 containers and 600+ IRQs

Open
#746 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
114
Forks
44
Avg merge
1d 8h
Merged PRs (30d)
20

Description

### Bug Description

Starting balloons pod may fail due to exceeding container runtime's plugin_request_timeout that defaults to 2 seconds. The reason is that synchronization may take very long on a large server that has already 100+ containers running. Noticed when enabling IRQ controls in such a system.

### Expected behavior

Synchronization should handle 100 containers within the default timeout.

### Experienced behavior

It did not.

### Steps to reproduce the behavior

Deploy the balloons plugin with IRQ configuration enabled on a server with 600+ IRQs to be managed and 100 containers running. Optionally include CPU class configurations that configure frequencies with cpufreq or SST.

### Environment

- nri-plugins version: e8e29bff9619bdc6cbb075a2ab18620017cbe514 (v0.13.1 + balloons IRQ support)
- nri plugin(s) running: balloons
- container runtime: containerd

### Additional context

Perhaps Synchronize should not apply system configurations (cpufreq, IRQ, ...) for each container separately, as long as there are containers left that may immediately make previous configuration invalid at least in some parts.

Contributor guide

Open the contributing guide

Research direction

Start with the balloons plugin's Synchronize() path and reproduce the issue using IRQ configuration with 600+ IRQs and about 100 containers, optionally including cpufreq or SST settings. Trace which synchronization work consumes the plugin_request_timeout and consider the issue complete when synchronization handles that workload within the default two-second timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.