epam / epam/cloud-pipeline

Auto-mount/unmount storages

Open
#3,976 0 comments 0 reactions 0 assignees View on GitHub
kind/enhancement
Dominant language
Java
Stars
163
Forks
65
Avg merge
11h 46m
Merged PRs (30d)
29

Description

**Background**
Currently, newly created storages that become available to a user do not automatically mount in already launched user runs. Similarly, storages that were mounted at the start of a run are not automatically unmounted if they are later deleted during the run.

**Approach**
Implement automatic mounting and unmounting of data storages during a user run:

- **Auto-mount**: New storages that become available to the user after the run has started should be automatically detected and mounted. Storages with changed mount options (permissions) during the run should be re-mount with updated options.
- **Auto-unmount**: Storages that were mounted at the start of the run but are later deleted should be automatically unmounted.

**Technical proposal**

1. Modify the [mount_storage.py](https://github.com/epam/cloud-pipeline/blob/develop/workflows/pipe-common/scripts/mount_storage.py) script to run as a daemon that continuously monitor available storages for the owner run user.
Periodically (based on `CP_CAP_MOUNT_REFRESH_INTERVAL`, default: 5 minutes):
- Fetch the list of storages available to the user.
- Compare with the previously known set.
- Mount newly available storages.
- Unmount storages that have been removed or are no longer accessible.
- Remount storages if necessary when mount option changes are detected:
- permission changes
- mountDisabled status
- region settings changes

2. Combine the [watch_mount_shares.py](https://github.com/epam/cloud-pipeline/blob/develop/workflows/pipe-common/scripts/watch_mount_shares.py) script logic regarding to NFS status monitoring with [mount_storage.py](https://github.com/epam/cloud-pipeline/blob/develop/workflows/pipe-common/scripts/mount_storage.py) script to prevent duplication of code and API calls.

3. Add feature flags to control which parts of the storage monitor are active (default: enable all):
- auto mount/umount
- auto permissions
- auto nfs status

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with workflows/pipe-common/scripts/mount_storage.py and workflows/pipe-common/scripts/watch_mount_shares.py, then trace how these scripts run during a user run. Compare the existing storage and NFS monitoring behavior with the proposed refresh interval and feature flags. Done means one coordinated monitor handles auto-mount, auto-unmount, permission changes, mountDisabled and region changes, plus NFS status monitoring.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.