common-workflow-language / common-workflow-language/cwltool

Provenance can not be used with parallel

Open
#1,059 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
Provenance should save the Research Object to a directory while `--parallel` option allows workflow steps to run in parallel.

## Actual Behavior
When `--provenance` is used at the same time with `--parallel`, a `WorkflowException` was raised.

## Workflow Code
Using the User Guide docker example as s testing case

```
/data/test$ cat docker.cwl
#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
baseCommand: node
hints:
DockerRequirement:
dockerPull: node:slim
inputs:
src:
type: File
inputBinding:
position: 1
outputs:
example_out:
type: stdout
stdout: output.txt
```

```
/data/test$ cat docker.cwl
#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
baseCommand: node
hints:
DockerRequirement:
dockerPull: node:slim
inputs:
src:
type: File
inputBinding:
position: 1
outputs:
example_out:
type: stdout
stdout: output.txt
zy2414@ld-mjeste19:/data/test$ cat docker-job.yml
src:
class: File
path: hello.js
```
```
/data/test$ cat hello.js
console.log("Hello World");
```

## Full Traceback
```pytb
/data/test$ cwl-runner --provenance=./prov --parallel --debug docker.cwl docker-job.yml
/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/bin/cwl-runner 1.0.20180820141117
[provenance] Temporary research object: /tmp/tmpw8fa06h0
Resolved 'docker.cwl' to 'file:///data/test/docker.cwl'
[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha1.txt: a3098e0e23eea9dff74193eff8ff018cb7855013 workflow/packed.cwl

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha256.txt: 84551d43ae61938cb154af874ae4f2b9cec7dffa9ea72e1a6523f78b5786d725 workflow/packed.cwl

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha512.txt: 60562e7c183c48ee2510f6fa1c1fac1d1b42283fc6b0dfbb8e67164894ae49674d16dfad9707834f0f142a007fde748f06ff803d90c323058c8321bb364512e2 workflow/packed.cwl

[provenance] Added packed workflow: workflow/packed.cwl
[job docker.cwl] initializing from file:///data/test/docker.cwl
[job docker.cwl] {
"src": {
"class": "File",
"location": "file:///data/test/hello.js",
"size": 28,
"basename": "hello.js",
"nameroot": "hello",
"nameext": ".js"
}
}
[job docker.cwl] path mappings is {
"file:///data/test/hello.js": [
"/data/test/hello.js",
"/var/lib/cwl/stg1816c3ce-ee89-4e60-80fc-b2ef9aa78dea/hello.js",
"File",
true
]
}
[job docker.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "node"
},
{
"position": [
1,
"src"
],
"datum": {
"class": "File",
"location": "file:///data/test/hello.js",
"size": 28,
"basename": "hello.js",
"nameroot": "hello",
"nameext": ".js",
"path": "/var/lib/cwl/stg1816c3ce-ee89-4e60-80fc-b2ef9aa78dea/hello.js",
"dirname": "/var/lib/cwl/stg1816c3ce-ee89-4e60-80fc-b2ef9aa78dea"
}
}
]
[job docker.cwl] /tmp/tmp63zrieun$ docker \
run \
-i \
--volume=/tmp/tmp63zrieun:/var/spool/cwl:rw \
--volume=/tmp/tmpi7mdlsoi:/tmp:rw \
--volume=/data/test/hello.js:/var/lib/cwl/stg1816c3ce-ee89-4e60-80fc-b2ef9aa78dea/hello.js:ro \
--workdir=/var/spool/cwl \
--read-only=true \
--log-driver=none \
--user=201808:700 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/var/spool/cwl \
node:slim \
node \
/var/lib/cwl/stg1816c3ce-ee89-4e60-80fc-b2ef9aa78dea/hello.js > /tmp/tmp63zrieun/output.txt
Got workflow error
Traceback (most recent call last):
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/executors.py", line 241, in runner
my_job.run(my_runtime_context)
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/job.py", line 496, in run
self._execute(runtime, env, runtimeContext)
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/job.py", line 235, in _execute
assert runtimeContext.prov_obj
AssertionError
Workflow error:
Traceback (most recent call last):
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/main.py", line 713, in main
logger=_logger)
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/executors.py", line 40, in __call__
return self.execute(*args, **kwargs)
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/executors.py", line 90, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/executors.py", line 306, in run_jobs
self.wait_for_next_completion(runtime_context)
File "/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/lib/python3.7/site-packages/cwltool/executors.py", line 272, in wait_for_next_completion
raise self.exceptions[0]
cwltool.errors.WorkflowException
[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha1.txt: e0e3f36fc62796ce057de426088b3105f0e46fe0 snapshot/docker.cwl

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha256.txt: 777f24f938100a86ef3de98cd58398f4c839c943369295003eb6d52bdffb55cd snapshot/docker.cwl

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha512.txt: 5abb2d0117e8619e92751e5817a890ab0da18aaa0196c89b4d2cf37d62467e88254a023ba300677e8b4ea83298386baf3a734f59d562d360d643c7e3be455f36 snapshot/docker.cwl

[provenance] Finalizing Research Object
[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha1.txt: f5435aecf55c6dd267d799af2b25a895a74d1c49 metadata/manifest.json

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha256.txt: e3f8df36ef0cfe7c102534f9c61e3d425c97ce300c1f88caf165fb371fbe9811 metadata/manifest.json

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha512.txt: 2993efeae3c48930f9800a2dec73b69dee3694fbae080c3a9370114a0358df858ff26937856ca18e0b6c704f6473ee4965f548d35726ea07a25241cbaed2f074 metadata/manifest.json

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha1.txt: 60f72b7c0a459402440d758da0eec022b9eb9c90 bag-info.txt

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha256.txt: 2d36ef7054d70e51f351da53c3665c2407a87b89ad2a2f72b3a51e7d4c5c0cf1 bag-info.txt

[provenance] Added to /tmp/tmpw8fa06h0/tagmanifest-sha512.txt: 620f5f7835e85821beecec9f1b3cc062ba1ce5914e1a21ae53854497c4e147b61fb1e1f0e1dc01d20fada4a2cd781a75ec874f7b1e839d6f427d5743aa1c68cc bag-info.txt

[provenance] Generated bagit metadata: /tmp/tmpw8fa06h0
[provenance] Deleting existing /data/test/prov
[provenance] Research Object saved to /data/test/prov
```

## Your Environment
* cwltool version: 1.0.20180820141117

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.