common-workflow-language / common-workflow-language/cwltool
InitialWorkDirRequirement `Error collecting output` when using DockerRequirement
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
When I do not use a `DockerRequirement`, `InitialWorkDirRequirement` completes successfully, but it fails when I introduce a minimal docker.
For example:
```
$ touch test.bam
$ cwltool --debug ~/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml --INPUT test.bam
/home/ubuntu/.virtualenvs/cwl/bin/cwltool 1.0
Parsed job order from command line: {
"INPUT": {
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"id": "/home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml",
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] initializing from file:///home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml
[job unix_initialworkdirrequirement.cwl.yaml] {
"INPUT": {
"basename": "test.bam",
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] path mappings is {
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/var/lib/cwl/stg5780b06e-07ad-4cbf-8e7a-5ed79b1d9854/test.bam",
"File"
]
}
[job unix_initialworkdirrequirement.cwl.yaml] command line bindings is [
{
"position": [
-1000000,
0
],
"valueFrom": "touch"
},
{
"position": [
0,
0
],
"valueFrom": null,
"do_eval": "$(inputs.INPUT.nameroot + \".bai\")"
}
]
[job unix_initialworkdirrequirement.cwl.yaml] /tmp/tmpMp0yPl$ docker \
run \
-i \
--volume=/home/ubuntu/test.bam:/var/lib/cwl/stg5780b06e-07ad-4cbf-8e7a-5ed79b1d9854/test.bam:ro \
--volume=/tmp/tmpMp0yPl:/var/spool/cwl:rw \
--volume=/tmp/tmpLHelpX:/tmp:rw \
--workdir=/var/spool/cwl \
--read-only=true \
--user=1000 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/var/spool/cwl \
alpine \
touch \
test.bai
[job unix_initialworkdirrequirement.cwl.yaml] initial work dir {
"_:667509cf-7b54-49a0-a24f-36faaa00cbdf": [
"_:667509cf-7b54-49a0-a24f-36faaa00cbdf",
"/tmp/tmpMp0yPl",
"Directory"
],
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/tmp/tmpMp0yPl/test.bam",
"File"
]
}
Error collecting output for parameter 'OUTPUT'
Traceback (most recent call last):
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/draft2tool.py", line 365, in collect_output_ports
ret[fragment] = self.collect_output(port, builder, outdir)
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/draft2tool.py", line 409, in collect_output
getListing(builder.fs_access, files)
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/process.py", line 162, in getListing
for ld in fs_access.listdir(loc):
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/stdfsaccess.py", line 31, in listdir
return [abspath(l, fn) for l in os.listdir(self._abs(fn))]
OSError: [Errno 2] No such file or directory: '/tmp/tmpMp0yPl/test.bam'
Error while running job: Error collecting output for parameter 'OUTPUT': [Errno 2] No such file or directory: '/tmp/tmpMp0yPl/test.bam'
[job unix_initialworkdirrequirement.cwl.yaml] completed permanentFail
[job unix_initialworkdirrequirement.cwl.yaml] {}
Final process status is permanentFail
[job unix_initialworkdirrequirement.cwl.yaml] Removing input staging directory /tmp/tmpqMfeZ1
[job unix_initialworkdirrequirement.cwl.yaml] Removing temporary directory /tmp/tmpLHelpX
Workflow error, try again with --debug for more information:
Process status is ['permanentFail']
Traceback (most recent call last):
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/main.py", line 707, in main
**vars(args))
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/main.py", line 218, in single_job_executor
raise WorkflowException(u"Process status is %s" % (final_status))
WorkflowException: Process status is ['permanentFail']
```
Below is the cwl.
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
requirements:
- class: DockerRequirement # Line 6
dockerPull: alpine # Line 7
- class: InitialWorkDirRequirement
listing:
- $(inputs.INPUT)
- class: InlineJavascriptRequirement
class: CommandLineTool
inputs:
- id: INPUT
type: File
outputs:
- id: OUTPUT
type: File
outputBinding:
glob: $(inputs.INPUT.basename)
secondaryFiles:
- ^.bai
arguments:
- valueFrom: $(inputs.INPUT.nameroot + ".bai")
position: 0
baseCommand: [touch]
```
If I comment out lines 6 & 7, the command completes:
```
(cwl)ubuntu@frontend001:~$ cwltool --debug ~/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml --INPUT test.bam
/home/ubuntu/.virtualenvs/cwl/bin/cwltool 1.0
Parsed job order from command line: {
"INPUT": {
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"id": "/home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml",
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] initializing from file:///home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml
[job unix_initialworkdirrequirement.cwl.yaml] {
"INPUT": {
"basename": "test.bam",
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] path mappings is {
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/tmp/tmp5h59nx/stg5d014b56-0a3a-4052-a325-06380cfb923d/test.bam",
"File"
]
}
[job unix_initialworkdirrequirement.cwl.yaml] command line bindings is [
{
"position": [
-1000000,
0
],
"valueFrom": "touch"
},
{
"position": [
0,
0
],
"valueFrom": null,
"do_eval": "$(inputs.INPUT.nameroot + \".bai\")"
}
]
[job unix_initialworkdirrequirement.cwl.yaml] /tmp/tmpAO2Yv9$ touch \
test.bai
[job unix_initialworkdirrequirement.cwl.yaml] initial work dir {
"_:5b8b2c9b-7cc6-494c-935a-21c8d11b6cbd": [
"_:5b8b2c9b-7cc6-494c-935a-21c8d11b6cbd",
"/tmp/tmpAO2Yv9",
"Directory"
],
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/tmp/tmpAO2Yv9/test.bam",
"File"
]
}
[job unix_initialworkdirrequirement.cwl.yaml] completed success
[job unix_initialworkdirrequirement.cwl.yaml] {
"OUTPUT": {
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"basename": "test.bam",
"location": "file:///tmp/tmpAO2Yv9/test.bam",
"secondaryFiles": [
{
"basename": "test.bai",
"location": "file:///tmp/tmpAO2Yv9/test.bai",
"class": "File"
}
],
"class": "File",
"size": 0
}
}
Final process status is success
[job unix_initialworkdirrequirement.cwl.yaml] Removing input staging directory /tmp/tmp5h59nx
[job unix_initialworkdirrequirement.cwl.yaml] Removing temporary directory /tmp/tmp939nLm
Moving /tmp/tmpAO2Yv9/test.bai to /home/ubuntu/test.bai
{
"OUTPUT": {
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"basename": "test.bam",
"location": "file:///home/ubuntu/test.bam",
"path": "/home/ubuntu/test.bam",
"secondaryFiles": [
{
"path": "/home/ubuntu/test.bai",
"basename": "test.bai",
"location": "file:///home/ubuntu/test.bai",
"class": "File"
}
],
"class": "File",
"size": 0
}
}
```
Contributor guide
Assessment
This issue has not been assessed yet.