nextflow-io / nextflow-io/nextflow
Feature request: Allow use of arbitrary network filesystems with the AWS batch executor
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
New feature
Currently, executor 'awsbatch' requires the use of a bucketDir / -bucket-dir. It looks like there's built-in support for Fusion, and a licenseable extenstion for FSx-for-Lustre and AWS EFS. It would be nice to be able to use any network filesystem with AWS batch (perhaps with less-rich support in the open-source version of nextflow).
Use case
We have a network filesystem that's none of the above (FlexFS) that's mounted in our AWS batch templates, and would like to use it with the nextflow awsbatch executor.
Suggested implementation
Add a setting for the awsbatch executor to skip the check that a -bucket-dir / bucketDir is defined, and in that case simply copy/link/etc. files into the standard workDir to stage them (assuming the workDir is on some kind of network filesystem).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the awsbatch executor's bucketDir validation and its work-file staging path. Compare the mentioned Fusion and FSx-for-Lustre/AWS EFS support, then define how an optional no-bucket-dir mode should stage files on a network filesystem; done means AWS Batch can use the configured workDir without requiring bucketDir.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, groovy
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100