common-workflow-language / common-workflow-language/cwltool

InitialWorkDirRequirement `Error collecting output` when using DockerRequirement

Đang mở
#126 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
376
Fork
255
Merge trung bình
2 ngày 7 giờ
Pull request đã merge (30 ngày)
12

Mô tả

When I do not use a `DockerRequirement`, `InitialWorkDirRequirement` completes successfully, but it fails when I introduce a minimal docker.

For example:

```
$ touch test.bam
$ cwltool --debug ~/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml --INPUT test.bam
/home/ubuntu/.virtualenvs/cwl/bin/cwltool 1.0
Parsed job order from command line: {
"INPUT": {
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"id": "/home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml",
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] initializing from file:///home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml
[job unix_initialworkdirrequirement.cwl.yaml] {
"INPUT": {
"basename": "test.bam",
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] path mappings is {
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/var/lib/cwl/stg5780b06e-07ad-4cbf-8e7a-5ed79b1d9854/test.bam",
"File"
]
}
[job unix_initialworkdirrequirement.cwl.yaml] command line bindings is [
{
"position": [
-1000000,
0
],
"valueFrom": "touch"
},
{
"position": [
0,
0
],
"valueFrom": null,
"do_eval": "$(inputs.INPUT.nameroot + \".bai\")"
}
]
[job unix_initialworkdirrequirement.cwl.yaml] /tmp/tmpMp0yPl$ docker \
run \
-i \
--volume=/home/ubuntu/test.bam:/var/lib/cwl/stg5780b06e-07ad-4cbf-8e7a-5ed79b1d9854/test.bam:ro \
--volume=/tmp/tmpMp0yPl:/var/spool/cwl:rw \
--volume=/tmp/tmpLHelpX:/tmp:rw \
--workdir=/var/spool/cwl \
--read-only=true \
--user=1000 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/var/spool/cwl \
alpine \
touch \
test.bai
[job unix_initialworkdirrequirement.cwl.yaml] initial work dir {
"_:667509cf-7b54-49a0-a24f-36faaa00cbdf": [
"_:667509cf-7b54-49a0-a24f-36faaa00cbdf",
"/tmp/tmpMp0yPl",
"Directory"
],
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/tmp/tmpMp0yPl/test.bam",
"File"
]
}
Error collecting output for parameter 'OUTPUT'
Traceback (most recent call last):
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/draft2tool.py", line 365, in collect_output_ports
ret[fragment] = self.collect_output(port, builder, outdir)
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/draft2tool.py", line 409, in collect_output
getListing(builder.fs_access, files)
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/process.py", line 162, in getListing
for ld in fs_access.listdir(loc):
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/stdfsaccess.py", line 31, in listdir
return [abspath(l, fn) for l in os.listdir(self._abs(fn))]
OSError: [Errno 2] No such file or directory: '/tmp/tmpMp0yPl/test.bam'
Error while running job: Error collecting output for parameter 'OUTPUT': [Errno 2] No such file or directory: '/tmp/tmpMp0yPl/test.bam'
[job unix_initialworkdirrequirement.cwl.yaml] completed permanentFail
[job unix_initialworkdirrequirement.cwl.yaml] {}
Final process status is permanentFail
[job unix_initialworkdirrequirement.cwl.yaml] Removing input staging directory /tmp/tmpqMfeZ1
[job unix_initialworkdirrequirement.cwl.yaml] Removing temporary directory /tmp/tmpLHelpX
Workflow error, try again with --debug for more information:
Process status is ['permanentFail']
Traceback (most recent call last):
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/main.py", line 707, in main
**vars(args))
File "/home/ubuntu/.virtualenvs/cwl/local/lib/python2.7/site-packages/cwltool/main.py", line 218, in single_job_executor
raise WorkflowException(u"Process status is %s" % (final_status))
WorkflowException: Process status is ['permanentFail']
```

Below is the cwl.

```
#!/usr/bin/env cwl-runner

cwlVersion: v1.0

requirements:
- class: DockerRequirement # Line 6
dockerPull: alpine # Line 7
- class: InitialWorkDirRequirement
listing:
- $(inputs.INPUT)
- class: InlineJavascriptRequirement

class: CommandLineTool

inputs:
- id: INPUT
type: File

outputs:
- id: OUTPUT
type: File
outputBinding:
glob: $(inputs.INPUT.basename)
secondaryFiles:
- ^.bai

arguments:
- valueFrom: $(inputs.INPUT.nameroot + ".bai")
position: 0

baseCommand: [touch]
```

If I comment out lines 6 & 7, the command completes:

```
(cwl)ubuntu@frontend001:~$ cwltool --debug ~/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml --INPUT test.bam
/home/ubuntu/.virtualenvs/cwl/bin/cwltool 1.0
Parsed job order from command line: {
"INPUT": {
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"id": "/home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml",
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] initializing from file:///home/ubuntu/cocleaning-cwl/tools/unix_initialworkdirrequirement.cwl.yaml
[job unix_initialworkdirrequirement.cwl.yaml] {
"INPUT": {
"basename": "test.bam",
"class": "File",
"location": "file:///home/ubuntu/test.bam"
},
"job_order": null
}
[job unix_initialworkdirrequirement.cwl.yaml] path mappings is {
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/tmp/tmp5h59nx/stg5d014b56-0a3a-4052-a325-06380cfb923d/test.bam",
"File"
]
}
[job unix_initialworkdirrequirement.cwl.yaml] command line bindings is [
{
"position": [
-1000000,
0
],
"valueFrom": "touch"
},
{
"position": [
0,
0
],
"valueFrom": null,
"do_eval": "$(inputs.INPUT.nameroot + \".bai\")"
}
]
[job unix_initialworkdirrequirement.cwl.yaml] /tmp/tmpAO2Yv9$ touch \
test.bai
[job unix_initialworkdirrequirement.cwl.yaml] initial work dir {
"_:5b8b2c9b-7cc6-494c-935a-21c8d11b6cbd": [
"_:5b8b2c9b-7cc6-494c-935a-21c8d11b6cbd",
"/tmp/tmpAO2Yv9",
"Directory"
],
"file:///home/ubuntu/test.bam": [
"/home/ubuntu/test.bam",
"/tmp/tmpAO2Yv9/test.bam",
"File"
]
}
[job unix_initialworkdirrequirement.cwl.yaml] completed success
[job unix_initialworkdirrequirement.cwl.yaml] {
"OUTPUT": {
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"basename": "test.bam",
"location": "file:///tmp/tmpAO2Yv9/test.bam",
"secondaryFiles": [
{
"basename": "test.bai",
"location": "file:///tmp/tmpAO2Yv9/test.bai",
"class": "File"
}
],
"class": "File",
"size": 0
}
}
Final process status is success
[job unix_initialworkdirrequirement.cwl.yaml] Removing input staging directory /tmp/tmp5h59nx
[job unix_initialworkdirrequirement.cwl.yaml] Removing temporary directory /tmp/tmp939nLm
Moving /tmp/tmpAO2Yv9/test.bai to /home/ubuntu/test.bai
{
"OUTPUT": {
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"basename": "test.bam",
"location": "file:///home/ubuntu/test.bam",
"path": "/home/ubuntu/test.bam",
"secondaryFiles": [
{
"path": "/home/ubuntu/test.bai",
"basename": "test.bai",
"location": "file:///home/ubuntu/test.bai",
"class": "File"
}
],
"class": "File",
"size": 0
}
}
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.