common-workflow-language / common-workflow-language/cwltool

Presigned urls for files doesn't work when cwltool:overrides is also present in input.json

Open
#1,753 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior

Be able to place `cwltool:overrides` as a key to input json whilst also specifying a presigned url as a file location attribute

## Actual Behavior

By specifying cwltool overrides, a tool check appears to take place and renders the location attribute invalid.

```
INFO /home/alexiswl/mambaforge/envs/cwltool-latest/bin/cwltool 3.1.20221018083734
INFO Resolved 'workflow.cwl' to 'file:///e/Users/awluc/OneDrive/GitHub/UMCCR/cwl-ica-cli/output-zips/tabix-workflow__0.2.6/workflow.cwl'
ERROR Tool definition failed validation:
input.json:2:3: checking field `vcf_file`
input.json:4:5: Field `location` contains undefined reference to
`https://stratus-gds-aps2.s3.ap-southeast-2.amazonaws.com/be72ba79-8a89-4d98-e0b1-08d9d6ea2bf9/test_tabix/input_vcf/trio.2010_06.ychr.sites.vcf.gz?X-Amz-Expires=604800&response-content-disposition=attachment%3Bfilename%3D%22trio.2010_06.ychr.sites.vcf.gz%22&response-content-type=application%2Fx-gzip&x-userId=ica&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARFCPI2IG5QI3WUOB/20221023/ap-southeast-2/s3/aws4_request&X-Amz-Date=20221023T030923Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7e2d39f1eaa739846c95eb5df568a3d34f2d0618795a9e18d4caa99f8864856a`
```

## Workflow Code
```
cwlVersion: v1.1
class: Workflow

# Extensions
$namespaces:
s: https://schema.org/
$schemas:
- https://schema.org/version/latest/schemaorg-current-http.rdf

# Metadata
s:author:
class: s:Person
s:name: Alexis Lucattini
s:email: Alexis.Lucattini@umccr.org
s:identifier: https://orcid.org/0000-0001-9754-647X

# ID/Docs
id: tabix-workflow--0.2.6
label: tabix-workflow v(0.2.6)
doc: |
Just run the tabix tool

requirements:
InlineJavascriptRequirement: {}
ScatterFeatureRequirement: {}
MultipleInputFeatureRequirement: {}
StepInputExpressionRequirement: {}

inputs:
vcf_file:
label: vcf file
doc: |
The input vcf file to be indexed
type: File

steps:
run_tabix_step:
label: run tabix step
doc: |
Run tabix workflow
in:
vcf_file:
source: vcf_file
out:
- id: vcf_file_indexed
run: tools/tabix/0.2.6/tabix__0.2.6.cwl

outputs:
vcf_file_indexed:
label: vcf file indexed
doc: |
The indexed vcf file
type: File
outputSource: run_tabix_step/vcf_file_indexed
```

### tools/tabix/0.2.6/tabix__0.2.6.cwl

```yaml
cwlVersion: v1.1
class: CommandLineTool

# Extensions
$namespaces:
s: https://schema.org/
ilmn-tes: http://platform.illumina.com/rdf/ica/
$schemas:
- https://schema.org/version/latest/schemaorg-current-http.rdf

# TODO - rename this to index-vcf-file and place as custom tool
# TODO - OR abstract this tool and support gff, gtf or sam files as well.

# Metadata
s:author:
class: s:Person
s:name: Alexis Lucattini
s:email: Alexis.Lucattini@umccr.org
s:identifier: https://orcid.org/0000-0001-9754-647X

# ID/Docs
id: tabix--0.2.6
label: tabix v(0.2.6)
doc: |
Add an index to a vcf file, more info can be found [here](http://www.htslib.org/doc/tabix.html)

hints:
ResourceRequirement:
ilmn-tes:resources:
tier: standard
type: standard
size: medium
coresMin: 1
ramMin: 4000
DockerRequirement:
dockerPull: quay.io/biocontainers/tabix:1.11--hdfd78af_0

requirements:
InlineJavascriptRequirement: {}
InitialWorkDirRequirement:
listing:
- entryname: create_index.sh
entry: |
#!/usr/bin/env bash

# Set pipe failure
set -euo pipefail

# Copy file over
cp "$(inputs.vcf_file.path)" "$(inputs.vcf_file.basename)"

# Run tabix
tabix -p vcf "$(inputs.vcf_file.basename)"

baseCommand: ["bash", "create_index.sh"]

inputs:
vcf_file:
label: vcf file
doc: |
The input vcf file to be indexed
type: File
streamable: true

outputs:
vcf_file_indexed:
label: vcf file indexed
doc: |
The indexed vcf file
type: File
outputBinding:
glob: "$(inputs.vcf_file.basename)"
secondaryFiles:
- pattern: ".tbi"
required: false

successCodes:
- 0
```

## Full Traceback
```pytb
INFO /home/alexiswl/mambaforge/envs/cwltool-latest/bin/cwltool 3.1.20221018083734
INFO Resolved 'workflow.cwl' to 'file:///e/Users/awluc/OneDrive/GitHub/UMCCR/cwl-ica-cli/output-zips/tabix-workflow__0.2.6/workflow.cwl'
ERROR Tool definition failed validation:
input.json:2:3: checking field `vcf_file`
input.json:4:5: Field `location` contains undefined reference to
`https://stratus-gds-aps2.s3.ap-southeast-2.amazonaws.com/be72ba79-8a89-4d98-e0b1-08d9d6ea2bf9/test_tabix/input_vcf/trio.2010_06.ychr.sites.vcf.gz?X-Amz-Expires=604800&response-content-disposition=attachment%3Bfilename%3D%22trio.2010_06.ychr.sites.vcf.gz%22&response-content-type=application%2Fx-gzip&x-userId=ica&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARFCPI2IG5QI3WUOB/20221023/ap-southeast-2/s3/aws4_request&X-Amz-Date=20221023T030923Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7e2d39f1eaa739846c95eb5df568a3d34f2d0618795a9e18d4caa99f8864856a`
Traceback (most recent call last):
File "/home/alexiswl/mambaforge/envs/cwltool-latest/lib/python3.10/site-packages/cwltool/main.py", line 1097, in main
job_order_object, input_basedir, jobloader = load_job_order(
File "/home/alexiswl/mambaforge/envs/cwltool-latest/lib/python3.10/site-packages/cwltool/main.py", line 385, in load_job_order
resolve_overrides(job_order_object, ov_uri, tool_file_uri)
File "/home/alexiswl/mambaforge/envs/cwltool-latest/lib/python3.10/site-packages/cwltool/load_tool.py", line 654, in resolve_overrides
cwl_docloader.resolve_all(ret, ov_uri)
File "schema_salad/ref_resolver.py", line 965, in resolve_all
File "schema_salad/ref_resolver.py", line 1197, in validate_links
schema_salad.exceptions.ValidationException: input.json:2:3: checking field `vcf_file`
input.json:4:5: Field `location` contains undefined reference to
`https://stratus-gds-aps2.s3.ap-southeast-2.amazonaws.com/be72ba79-8a89-4d98-e0b1-08d9d6ea2bf9/test_tabix/input_vcf/trio.2010_06.ychr.sites.vcf.gz?X-Amz-Expires=604800&response-content-disposition=attachment%3Bfilename%3D%22trio.2010_06.ychr.sites.vcf.gz%22&response-content-type=application%2Fx-gzip&x-userId=ica&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARFCPI2IG5QI3WUOB/20221023/ap-southeast-2/s3/aws4_request&X-Amz-Date=20221023T030923Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7e2d39f1eaa739846c95eb5df568a3d34f2d0618795a9e18d4caa99f8864856a`

```

## Input json (that fails validation step as shown above)

```json
{
"vcf_file": {
"class": "File",
"location": "https://stratus-gds-aps2.s3.ap-southeast-2.amazonaws.com/be72ba79-8a89-4d98-e0b1-08d9d6ea2bf9/test_tabix/input_vcf/trio.2010_06.ychr.sites.vcf.gz?X-Amz-Expires=604800&response-content-disposition=attachment%3Bfilename%3D%22trio.2010_06.ychr.sites.vcf.gz%22&response-content-type=application%2Fx-gzip&x-userId=ica&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARFCPI2IG5QI3WUOB/20221023/ap-southeast-2/s3/aws4_request&X-Amz-Date=20221023T030923Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7e2d39f1eaa739846c95eb5df568a3d34f2d0618795a9e18d4caa99f8864856a"
},
"cwltool:overrides": {
"workflow.cwl#tabix-workflow--0.2.6/run_tabix_step": {
"requirements": {
"DockerRequirement": {
"dockerPull": "public.ecr.aws/biocontainers/tabix:0.2.6--ha92aebf_0"
}
}
}
}
}
```

## Input json that passes validation (and workflow successfully completes)

```
{
"vcf_file": {
"class": "File",
"location": "https://stratus-gds-aps2.s3.ap-southeast-2.amazonaws.com/be72ba79-8a89-4d98-e0b1-08d9d6ea2bf9/test_tabix/input_vcf/trio.2010_06.ychr.sites.vcf.gz?X-Amz-Expires=604800&response-content-disposition=attachment%3Bfilename%3D%22trio.2010_06.ychr.sites.vcf.gz%22&response-content-type=application%2Fx-gzip&x-userId=ica&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARFCPI2IG5QI3WUOB/20221023/ap-southeast-2/s3/aws4_request&X-Amz-Date=20221023T030923Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7e2d39f1eaa739846c95eb5df568a3d34f2d0618795a9e18d4caa99f8864856a"
}
}
```

## Workaround

Specify overrides in an overrides file instead

```json
{
"cwltool:overrides": {
"workflow.cwl#tabix-workflow--0.2.6/run_tabix_step": {
"requirements": {
"DockerRequirement": {
"dockerPull": "public.ecr.aws/biocontainers/tabix:0.2.6--ha92aebf_0"
}
}
}
}
}
```

```bash
cwltool --podman --overrides overrides.json workflow.cwl input.json
```

## Your Environment
* cwltool version:
Check using ``cwltool --version``
```
cwltool --version
/home/alexiswl/mambaforge/envs/cwltool-latest/bin/cwltool 3.1.20221018083734
```

## Additional notes

Input file was downloaded from ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/pilot_data/release/2010_07/trio/snps/trio.2010_06.ychr.sites.vcf.gz and then uploaded onto our [ICAv2 platform](https://help.ica.illumina.com/) where I generated a presigned url. This will expire in a week, please let me know if you need a new one for reproducibility.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.