common-workflow-language / common-workflow-language/cwltool

Tool fails with "path is outside the designated output directory" if input has secondaryFiles and using InitialWorkDirRequirement

Open
#1,349 5 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
When staging input file to the output directory the secondaryFiles should be also staged into the same directory, otherwise tools fails to execute with "path is outside the designated output directory" error when mounting files to the docker container.

## Actual Behavior
Fails with "path is outside the designated output directory"

## Workflow Code
Failure example
```
git clone https://github.com/Barski-lab/workflows.git --recursive
cd workflows
git checkout c2591a0d550001f12ecc97c1d54f3659e20eff82
cd ..
cwltool --debug ./workflows/tools/samtools-rmdup.cwl ./workflows/tests/samtools-rmdup-1.json
```
Successful example
```
git clone https://github.com/Barski-lab/workflows.git --recursive
cd workflows
git checkout c2591a0d550001f12ecc97c1d54f3659e20eff82
cd ..
cwltool --debug ./workflows/tools/samtools-rmdup.cwl ./workflows/tests/samtools-rmdup-2.json
```

## Full Traceback
```pytb
cwltool --debug ~/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl ~/workspaces/cwl_ws/workflows/tests/samtools-rmdup-1.json
INFO /Users/tester/venv/cwl_airflow_devel/bin/cwltool 3.0.20200807132242
INFO Resolved '/Users/tester/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl' to 'file:///Users/tester/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl'
WARNING http://schema.org/docs/!DOCTYPE html does not look like a valid URI, trying to serialize this will break.
WARNING html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" does not look like a valid URI, trying to serialize this will break.
/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/rdflib/plugins/parsers/structureddata.py:29: UserWarning: html5lib not found! RDFa and Microdata parsers will not be available.
'html5lib not found! RDFa and Microdata ' +
Could not load extension schema http://schema.org/docs/schema_org_rdfa.html: html5lib is not installed, cannot use RDFa and Microdata parsers.
DEBUG [job samtools-rmdup.cwl] initializing from file:///Users/tester/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl
DEBUG [job samtools-rmdup.cwl] {
"bam_file": {
"class": "File",
"location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
"secondaryFiles": [
{
"class": "File",
"location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
"size": 1368,
"basename": "chr4_100_mapped_reads.Aligned.out.bam.bai",
"nameroot": "chr4_100_mapped_reads.Aligned.out.bam",
"nameext": ".bai"
}
],
"size": 11034,
"basename": "chr4_100_mapped_reads.Aligned.out.bam",
"nameroot": "chr4_100_mapped_reads.Aligned.out",
"nameext": ".bam"
},
"single_end": false,
"bash_script": "#!/bin/bash\nif [ \"$0\" = \"true\" ]\nthen\n samtools rmdup \"${@:1}\"\nelse\n echo \"Skip samtools rmdup \" ${@:1}\n mv ${@: -2}\nfi\n",
"output_filename": "",
"trigger": true,
"force_single_end": null
}
DEBUG [job samtools-rmdup.cwl] path mappings is {
"file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam": [
"/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
"/VUCcrH/input_file_backup",
"WritableFile",
false
],
"file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai": [
"/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
"/VUCcrH/chr4_100_mapped_reads.Aligned.out.bam.bai",
"File",
false
]
}
DEBUG [job samtools-rmdup.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "bash"
},
{
"position": [
-1000000,
1
],
"datum": "-c"
},
{
"position": [
1,
"bash_script"
],
"datum": "#!/bin/bash\nif [ \"$0\" = \"true\" ]\nthen\n samtools rmdup \"${@:1}\"\nelse\n echo \"Skip samtools rmdup \" ${@:1}\n mv ${@: -2}\nfi\n"
},
{
"position": [
2,
"trigger"
],
"valueFrom": "${ return self ? \"true\" : \"false\" }\n",
"datum": true
},
{
"position": [
3,
"single_end"
],
"prefix": "-s",
"datum": false
},
{
"position": [
10,
"bam_file"
],
"datum": {
"class": "File",
"location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
"secondaryFiles": [
{
"class": "File",
"location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
"size": 1368,
"basename": "chr4_100_mapped_reads.Aligned.out.bam.bai",
"nameroot": "chr4_100_mapped_reads.Aligned.out.bam",
"nameext": ".bai",
"path": "/VUCcrH/chr4_100_mapped_reads.Aligned.out.bam.bai",
"dirname": "/VUCcrH"
}
],
"size": 11034,
"basename": "input_file_backup",
"nameroot": "input_file_backup",
"nameext": "",
"path": "/VUCcrH/input_file_backup",
"dirname": "/VUCcrH"
}
},
{
"position": [
11,
"output_filename"
],
"valueFrom": "${\n if (self == \"\" || inputs.trigger == false){\n return default_output_filename();\n } else {\n return self;\n }\n}\n",
"datum": ""
},
{
"valueFrom": "${\n if (inputs.output_filename == \"\" || inputs.trigger == false){\n return \" > \" + default_output_filename().split('.').slice(0,-1).join('.') + \"_samtools_rmdup_report.txt 2>&1\";\n } else {\n return \" > \" + inputs.output_filename.split('.').slice(0,-1).join('.') + \"_samtools_rmdup_report.txt 2>&1\";\n }\n}\n",
"position": [
100000,
0
],
"shellQuote": false
}
]
DEBUG [job samtools-rmdup.cwl] initial work dir {
"file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam": [
"/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
"/VUCcrH/input_file_backup",
"WritableFile",
true
],
"file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai": [
"/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
"/var/lib/cwl/stg4f8de60a-bcda-4f9e-9d35-e83ba82755c1/chr4_100_mapped_reads.Aligned.out.bam.bai",
"WritableFile",
true
]
}
ERROR Workflow error:
No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai', target='/var/lib/cwl/stg4f8de60a-bcda-4f9e-9d35-e83ba82755c1/chr4_100_mapped_reads.Aligned.out.bam.bai', type='WritableFile', staged=True)
Traceback (most recent call last):
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/main.py", line 1132, in main
tool, initialized_job_order_object, runtimeContext, logger=_logger
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/executors.py", line 59, in __call__
return self.execute(process, job_order_object, runtime_context, logger)
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/executors.py", line 150, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/executors.py", line 257, in run_jobs
job.run(runtime_context)
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/job.py", line 828, in run
(runtime, cidfile) = self.create_runtime(env, runtimeContext)
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/docker.py", line 373, in create_runtime
tmpdir_prefix=runtimeContext.tmpdir_prefix,
File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/job.py", line 700, in add_volumes
"$(runtime.outdir): {}".format(vol)
cwltool.errors.WorkflowException: No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai', target='/var/lib/cwl/stg4f8de60a-bcda-4f9e-9d35-e83ba82755c1/chr4_100_mapped_reads.Aligned.out.bam.bai', type='WritableFile', staged=True)
```

## Your Environment
cwltool==3.0.20200807132242
schema-salad==7.0.20200811075006

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.