nextflow-io / nextflow-io/nextflow

AWS profile not picked up when using Nextflow Fusion

Open
#4,868 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug storage/aws storage/fusion
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

When I run a Nextflow workflow with Nextflow Fusion (input is on S3, output is on S3, work dir is on S3), Nextflow doesn't seem to pick up my AWS_PROFILE environment variable when resolving the work directory.

script.sh:

#!/bin/bash

# set AWS profile
export AWS_PROFILE=di

# create config
cat > /tmp/nextflow.config <<EOF
fusion {
    enabled = true
    exportStorageCredentials = true
}

wave {
    enabled = true
}
EOF

# run component
DATA=s3://data-intuitive-tmp/test-nextflow-wave-fusion/resources

NXF_VER=23.10.1 nextflow run \
  viash-io/test-nextflow-wave-fusion \
  -r main_build \
  -main-script target/nextflow/method/main.nf \
  -profile docker \
  -latest \
  -w "s3://data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue" \
  -c /tmp/nextflow.config \
  --input $DATA/input1.txt \
  --multiple_input "$DATA/input1.txt;$DATA/input2.txt" \
  --publish_dir "s3://data-intuitive-tmp/test-nextflow-wave-fusion/output/github_issue"
Script output 
N E X T F L O W  ~  version 23.10.1
Pulling viash-io/test-nextflow-wave-fusion ...
 Already-up-to-date
Launching `https://github.com/viash-io/test-nextflow-wave-fusion` [elegant_linnaeus] DSL2 - revision: 33e8c0898e [main_build]
executor >  local (fusion enabled) (1)
[0c/11dfda] process > method:processWf:method_process (run)          [  0%] 0 of 1
[-        ] process > method:publishStatesSimpleWf:publishStatesProc -
ERROR ~ Error executing process > 'method:processWf:method_process (run)'

Caused by:
  Process `method:processWf:method_process (run)` terminated with an error exit status (127)

Command executed:
  ...

Command exit status:
  127

Command output:
  (empty)

Command error:
  10:42AM ERR reading from data store error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'" path=/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339
  10:42AM WRN timeout waiting for FUSE mount
  10:42AM INF shutdown filesystem start
  10:42AM INF shutdown filesystem done
  10:42AM WRN using anonymous credentials to connect S3 region=us-east-1
  10:42AM INF Not mounting S3 data store - operation error S3: PutObject, https response error StatusCode: 403, RequestID: HZSVEPCTVBNBXTW5, HostID: IIc0G2vr2I4+90ewrfvO2wEB4kMGyyJZbPkqJRLmXEkFf3l0RU4q/okdmE2AUiT4bNm5XfYnMlg=, api error AccessDenied: Access Denied
  10:42AM ERR no datastore error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'"
  10:42AM ERR reading from data f0c2c7008339'"
  10:42AM ERR reading from data store error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'" path=/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339
  10:43AM WRN timeout waiting for FUSE mount
  10:43AM INF shutdown filesystem start
  10:43AM INF shutdown filesystem done
  10:43AM WRN using anonymous credentials to connect S3 region=us-east-1
  10:43AM INF Not mounting S3 data store - operation error S3: PutObject, https response error StatusCode: 403, RequestID: YGMB6Z2H9YG2N99K, HostID: WmBTHjaYp6BmjhCH7Fogk997vEUMZWwq1iCCdv02AZrRE0NzgeWM1j9vAZZQ5QNlCRpXBXXg10cVSJ5kd3fNRQ==, api error AccessDenied: Access Denied
  10:43AM ERR no datastore error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'"
  10:43AM ERR reading from data store error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'" path=/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339
  10:43AM WRN timeout waiting for FUSE mount
  10:43AM INF shutdown filesystem start
  10:43AM INF shutdown filesystem done
  10:43AM ERR creating .command.log file error="open /fusion/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339/.command.log: no such file or directory"
  bash: /fusion/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339/.command.run: No such file or directory
  10:43AM INF shutdown filesystem start
  10:43AM ERR on FUSE sigterm send error="os: process already finished"
  10:43AM INF shutdown filesystem done
  10:43AM WRN using anonymous credentials to connect S3 region=us-east-1
  10:43AM INF Not mounting S3 data store - operation error S3: PutObject, https response error StatusCode: 403, RequestID: XPJJAAJT1GP6DQHV, HostID: Quu+aMr9aQy5c9iGaE/cE14EOQAq5Cij4Tf7Wv3cidqykFf9i9SxRN0xRaQ9JSlpD9tXdMmgKNQ=, api error AccessDenied: Access Denied
  10:43AM ERR no datastore error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'"
  10:43AM ERR reading from data store error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'" path=/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339
  10:44AM WRN timeout waiting for FUSE mount
  10:44AM INF shutdown filesystem start
  10:44AM INF shutdown filesystem done
  10:44AM WRN using anonymous credentials to connect S3 region=us-east-1
  10:44AM INF Not mounting S3 data store - operation error S3: PutObject, https response error StatusCode: 403, RequestID: 3YS0EF7DCBZHM33Z, HostID: EJeDJUhQsIXrrzz4IK1V85LRITxhopiWyT3BEu63xBI5tqVS92HwOU8Vhiuwm0a625+7b2ILrUE=, api error AccessDenied: Access Denied
  10:44AM ERR no datastore error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'"
  10:44AM ERR reading from data store error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'" path=/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339
  10:44AM WRN timeout waiting for FUSE mount
  10:44AM INF shutdown filesystem start
  10:44AM INF shutdown filesystem done
  10:44AM WRN using anonymous credentials to connect S3 region=us-east-1
  10:44AM INF Not mounting S3 data store - operation error S3: PutObject, https response error StatusCode: 403, RequestID: 5WWA767GJWTWZ1MG, HostID: y1YAQN34aND22of5hFhwwOP4jZ6z3Yk11Tkb4l9xBbU1YSortDfqQA3ZAPJUHTj+35zitVN5Phk=, api error AccessDenied: Access Denied
  10:44AM ERR no datastore error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'"
  10:44AM ERR reading from data store error="unknown schema 's3' data store for path '/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339'" path=/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339
  10:45AM WRN timeout waiting for FUSE mount
  10:45AM INF shutdown filesystem start
  10:45AM INF shutdown filesystem done
  10:45AM ERR creating .command.log file error="open /fusion/s3/data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue/0c/11dfda6a9cfec968a2f0c2c7008339/.command.log: no such file or directo
executor >  local (fusion enabled) (1)
[0c/11dfda] process > method:processWf:method_process (run)          [100%] 1 of 1, failed: 1 ✘
[-        ] process > method:publishStatesSimpleWf:publishStatesProc -
Expected behavior and actual behavior

I would expect Nextflow to pick up my di AWS_PROFILE. I verified that my AWS PROFILE is indeed working, because when I don't use Fusion (input is on S3, output is on S3, work dir is local(!!)), Nextflow manages to fetch data from / publish data to my private S3 bucket:

script_nofusion.sh:

#!/bin/bash

# set AWS profile
export AWS_PROFILE=di

# run component
DATA=s3://data-intuitive-tmp/test-nextflow-wave-fusion/resources

NXF_VER=23.10.1 nextflow run \
  viash-io/test-nextflow-wave-fusion \
  -r main_build \
  -main-script target/nextflow/method/main.nf \
  -profile docker \
  -latest \
  --input $DATA/input1.txt \
  --multiple_input "$DATA/input1.txt;$DATA/input2.txt" \
  --publish_dir "s3://data-intuitive-tmp/test-nextflow-wave-fusion/output/github_issue"
Script output 
N E X T F L O W  ~  version 23.10.1
Pulling viash-io/test-nextflow-wave-fusion ...
 Already-up-to-date
Launching `https://github.com/viash-io/test-nextflow-wave-fusion` [jovial_ramanujan] DSL2 - revision: 33e8c0898e [main_build]
executor >  local (2)
[79/0affac] process > method:processWf:method_process (run)                [100%] 1 of 1 ✔
[68/5d6ea2] process > method:publishStatesSimpleWf:publishStatesProc (run) [100%] 1 of 1 ✔

When I manually define the aws.accessKey and the aws.secretKey in my nextflow config, the Nextflow workflow now works as intended:

script.sh:

#!/bin/bash

# do not set AWS profile
# export AWS_PROFILE=di

# create config
cat > /tmp/nextflow.config <<EOF
fusion {
    enabled = true
    exportStorageCredentials = true
}

aws {
    accessKey = "..."
    secretKey = "..."
}

wave {
    enabled = true
}
EOF

# run component
DATA=s3://data-intuitive-tmp/test-nextflow-wave-fusion/resources

NXF_VER=23.10.1 nextflow run \
  viash-io/test-nextflow-wave-fusion \
  -r main_build \
  -main-script target/nextflow/method/main.nf \
  -profile docker \
  -latest \
  -w "s3://data-intuitive-tmp/test-nextflow-wave-fusion/work/github_issue" \
  -c /tmp/nextflow.config \
  --input $DATA/input1.txt \
  --multiple_input "$DATA/input1.txt;$DATA/input2.txt" \
  --publish_dir "s3://data-intuitive-tmp/test-nextflow-wave-fusion/output/github_issue"
Script output 
N E X T F L O W  ~  version 23.10.1
Pulling viash-io/test-nextflow-wave-fusion ...
 Already-up-to-date
Launching `https://github.com/viash-io/test-nextflow-wave-fusion` [small_davinci] DSL2 - revision: 33e8c0898e [main_build]
executor >  local (fusion enabled) (2)
[a6/387fb4] process > method:processWf:method_process (run)                [100%] 1 of 1 ✔
[5e/ae32f3] process > method:publishStatesSimpleWf:publishStatesProc (run) [100%] 1 of 1 ✔
Steps to reproduce the problem

To reproduce this problem, you'll need access to a private S3 repository containing one or more files. This workflow simply copies the input file to the output, so you can use whatever file.

#!/bin/bash

# set AWS profile
export AWS_BUCKET=s3://path/to/your/s3
export AWS_PROFILE=your_aws_profile

# create config
cat > /tmp/nextflow.config <<EOF
fusion {
    enabled = true
    exportStorageCredentials = true
}

wave {
    enabled = true
}
EOF

# run component
DATA="$AWS_BUCKET/resources"

NXF_VER=23.10.1 nextflow run \
  viash-io/test-nextflow-wave-fusion \
  -r main_build \
  -main-script target/nextflow/method/main.nf \
  -profile docker \
  -latest \
  -w "$AWS_BUCKET/work" \
  -c /tmp/nextflow.config \
  --input "$AWS_BUCKET/resources/input1.txt" \
  --multiple_input "$AWS_BUCKET/resources/input1.txt;$AWS_BUCKET/resources/input2.txt" \
  --publish_dir "$AWS_BUCKET/output"

↑ change the AWS_BUCKET, AWS_PROFILE, --input, --multiple_input, and --publish_dir to point to a private S3 bucket with resources resources/input1.txt and resources/input2.txt.

Program output

See above.

.nextflow.log: .nextflow.log

Environment
  • Nextflow version: 23.10.2
  • Java version: 17.0.9
  • Operating system: Fedora Linux 39
  • Bash version: GNU bash, version 5.2.26(1)-release (x86_64-redhat-linux-gnu)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the issue's script.sh with Fusion enabled and compare it with script_nofusion.sh and the configuration in /tmp/nextflow.config. Trace how Fusion handles AWS_PROFILE versus explicit aws.accessKey and aws.secretKey values for the S3 work directory. Done means the reproduced workflow mounts and uses the private S3 work directory with AWS_PROFILE, without manually configured credentials.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.