Avoid excessive I/O and blocking when measuring scratch directory size
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
Currently the agent performs collection of performance metrics upon every 5 seconds, including recusive-sum of the file sizes in scratch directories used by session containers.
This may result in severe performance degradation and potential blocking of the default asyncio executor thread pool if there are many number of files in scratch directories, depending on the measurement interval, the number of CPU cores, the number of containers, and the number of scratch files.
Let's avoid such blocking situation and reduce the I/O load by prolonging the interval of measuring scratch directory size and applying filesystem-specific optimizations if available.
Ultimately we also need to refactor the statistics collection subsystem in agents because currently it performs many duplicate measurements due to the "abstraction wall" of per-container and per-node stats. We could separate measurement tasks and collection tasks sharing the same `StatContext` to avoid such duplication and apply different measurement intervals for different metrics.
JIRA Issue: BA-280
Contributor guide
Assessment
This issue has not been assessed yet.