lablup / lablup/backend.ai

Add a init/health check for mountpoints

Open
#1,639 3 comments 0 reactions 1 assignee Assigned to @kyujin-cho View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

There are many cases in production setups that filesystem mounts fail or get disconnected at runtinme. Also some high-performance NFS backends take a very long time (e.g., 1-2 minutes) to initialize their mounts when the server is rebooted.

Although we are dealing with this kind of issues by adding a busy-wait loop in the runner scripts or mangling some systemd configurations, but it is sometimes difficult to ensure the customers that it's _not_ our problems.

## Detection

### HOWTO

Let's use [mountpoint](https://man7.org/linux/man-pages/man1/mountpoint.1.html) to check/wait if the given set of directories are actually mounted or not upon startup.If they fail to mount for a long time (e.g., 10 minutes), let's make agents and storage-proxy to actively fail and show appropriate error logs.

## Mitigation

- Either:\* Explicitly reject kernel creation requests with vfolder mounts on lost volumes with proper error messages.
- Turn the agent's `schedulable` to off.

- Make an alarm to the cluster administrator.\* ...so that he/she could escalate the issue to the infrastructure team/manager.
- After the infra-level issue is resolved, we could instruct the administrator to turn on `schedulable` of the agent or restart the agent.

```Java
### Steps
- [ ] https://github.com/lablup/backend.ai/issues/1843
- [ ] https://github.com/lablup/backend.ai/issues/1845
- [ ] https://github.com/lablup/backend.ai/issues/1844
- [ ] https://github.com/lablup/backend.ai/issues/1846
- [ ] https://github.com/lablup/backend.ai/issues/1847
- [ ] https://github.com/lablup/backend.ai/issues/104
- [ ] https://github.com/lablup/backend.ai/issues/3055
```

## Expected result

This will greatly reduce the field support efforts that are related to storage issues.

JIRA Issue: BA-186

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.