FNNDSC / FNNDSC/pfcon

Create emphemeral volumes for inputdir

Open
#172 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
11
Forks
25
PR merge metrics
No merged PRs in 30d

Description

Currently, _pfcon_ copies the data of an `inputdir` from "`STOREBASE`", the persistent volume containing _CUBE_'s files, to a temporary `'key-' + job_id` directory. This temporary directory exists in the same volume as `STOREBASE`.

What if _pfcon_ creates an ephemeral volume for the `'key-' + job_id` directory instead of reusing the `STOREBASE`'s volume? This would enable some optimizations in multi-node Kubernetes:

- Ephemeral volumes for jobs can use a different [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/) from _CUBE_, optimizing performance. Heck, it could even use a memory-backed tmpfs or [RAM drive](https://en.wikipedia.org/wiki/RAM_drive).
- Some clusters lack the ability to provide ReadWriteMany volumes. If the system could copy data out from _CUBE_'s storage to ephemeral volumes, it would be possible to use ReadWriteOnce volumes and benefit from a mulit-node cluster.
- The "delete job" could be an atomic volume delete operation, instead of a `rm -rf`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.