I want to post-process a multi-TB dataset on a different HPC cluster than the one it was generated on
- Dominant language
- Python
- Stars
- 60
- Forks
- 18
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 38
Description
@CarolineCardinale's use case and initial idea:
1. Automatically chunk files into 500 GB gzip archives and download from cluster 1 to local disk
2. Upload chunks to cluster 2, uncompress, and analyze
Can this be made into a fully-automated pipeline though?
```yaml
environments:
cluster1:
kind: slurm
host: cluster.whatever.edu
cluster2:
kind: slurm
host: cluster.elsewhere.edu
my-conda-env:
kind: conda
path: environment.yml
pipeline:
stages:
run-sim:
kind: shell-script
script_path: run-my-sim.sh
environment: cluster1
outputs:
- path: results
kind: chunked-gzip
chunk_size: 500GB
post-process:
kind: python-script
script_path: scripts/postprocess.py
args:
- --time={time}
environment: cluster2:my-conda-env
inputs:
- results/time={time}.h5
iterate_over:
- arg_name: time
values:
- range: {start: 0, stop: 100, step: 1}
```
In this case, the `post-process` stage could download the gzipped chunk it needs for a given time, since we'd save an index based on the proposed `chunked-gzip` output kind. It would uncompress, do the work, then optionally delete after. This would also help with some of DVC's limitations regarding small files. For many TB of storage, however, users will probably need to setup their own storage. Google Drive is actually only [$100/mo for 20 TB](https://one.google.com/about/plans?g1_landing_page=0).
Contributor guide
Assessment
This issue has not been assessed yet.