lablup / lablup/backend.ai

Avoid excessive I/O and blocking when measuring scratch directory size

Open
#330 1 comment 0 reactions 1 assignee Claimed by @achimnol View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Currently the agent performs collection of performance metrics upon every 5 seconds, including recusive-sum of the file sizes in scratch directories used by session containers.

This may result in severe performance degradation and potential blocking of the default asyncio executor thread pool if there are many number of files in scratch directories, depending on the measurement interval, the number of CPU cores, the number of containers, and the number of scratch files.

Let's avoid such blocking situation and reduce the I/O load by prolonging the interval of measuring scratch directory size and applying filesystem-specific optimizations if available.

Ultimately we also need to refactor the statistics collection subsystem in agents because currently it performs many duplicate measurements due to the "abstraction wall" of per-container and per-node stats. We could separate measurement tasks and collection tasks sharing the same `StatContext` to avoid such duplication and apply different measurement intervals for different metrics.

JIRA Issue: BA-280

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.