Add a init/health check for mountpoints
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
There are many cases in production setups that filesystem mounts fail or get disconnected at runtinme. Also some high-performance NFS backends take a very long time (e.g., 1-2 minutes) to initialize their mounts when the server is rebooted.
Although we are dealing with this kind of issues by adding a busy-wait loop in the runner scripts or mangling some systemd configurations, but it is sometimes difficult to ensure the customers that it's _not_ our problems.
## Detection
### HOWTO
Let's use [mountpoint](https://man7.org/linux/man-pages/man1/mountpoint.1.html) to check/wait if the given set of directories are actually mounted or not upon startup.If they fail to mount for a long time (e.g., 10 minutes), let's make agents and storage-proxy to actively fail and show appropriate error logs.
## Mitigation
- Either:\* Explicitly reject kernel creation requests with vfolder mounts on lost volumes with proper error messages.
- Turn the agent's `schedulable` to off.
- Make an alarm to the cluster administrator.\* ...so that he/she could escalate the issue to the infrastructure team/manager.
- After the infra-level issue is resolved, we could instruct the administrator to turn on `schedulable` of the agent or restart the agent.
```Java
### Steps
- [ ] https://github.com/lablup/backend.ai/issues/1843
- [ ] https://github.com/lablup/backend.ai/issues/1845
- [ ] https://github.com/lablup/backend.ai/issues/1844
- [ ] https://github.com/lablup/backend.ai/issues/1846
- [ ] https://github.com/lablup/backend.ai/issues/1847
- [ ] https://github.com/lablup/backend.ai/issues/104
- [ ] https://github.com/lablup/backend.ai/issues/3055
```
## Expected result
This will greatly reduce the field support efforts that are related to storage issues.
JIRA Issue: BA-186
Contributor guide
Assessment
This issue has not been assessed yet.