nextflow-io / nextflow-io/nextflow

Feature request: Allow use of arbitrary network filesystems with the AWS batch executor

Open
#6,554 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

executor/aws-batch
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

Currently, executor 'awsbatch' requires the use of a bucketDir / -bucket-dir. It looks like there's built-in support for Fusion, and a licenseable extenstion for FSx-for-Lustre and AWS EFS. It would be nice to be able to use any network filesystem with AWS batch (perhaps with less-rich support in the open-source version of nextflow).

Use case

We have a network filesystem that's none of the above (FlexFS) that's mounted in our AWS batch templates, and would like to use it with the nextflow awsbatch executor.

Suggested implementation

Add a setting for the awsbatch executor to skip the check that a -bucket-dir / bucketDir is defined, and in that case simply copy/link/etc. files into the standard workDir to stage them (assuming the workDir is on some kind of network filesystem).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the awsbatch executor's bucketDir validation and its work-file staging path. Compare the mentioned Fusion and FSx-for-Lustre/AWS EFS support, then define how an optional no-bucket-dir mode should stage files on a network filesystem; done means AWS Batch can use the configured workDir without requiring bucketDir.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, groovy
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.