common-workflow-language / common-workflow-language/cwltool

Record container id failed when a relative path is passed to --cidfile-dir

Open
#1,058 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
Container ids saved to .cid files in a specified directory.

## Actual Behavior
When a relative directory is given to `--cidfile-dir`, docker failed to file the directory. Maybe `cwltool` should resolve to the absolute path then pass it to the docker command?

## Workflow Code
Using the User Guide docker example as testing case

```
/data/test$ cat docker.cwl
#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
baseCommand: node
hints:
DockerRequirement:
dockerPull: node:slim
inputs:
src:
type: File
inputBinding:
position: 1
outputs:
example_out:
type: stdout
stdout: output.txt
```

```
/data/test$ cat docker.cwl
#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
baseCommand: node
hints:
DockerRequirement:
dockerPull: node:slim
inputs:
src:
type: File
inputBinding:
position: 1
outputs:
example_out:
type: stdout
stdout: output.txt
zy2414@ld-mjeste19:/data/test$ cat docker-job.yml
src:
class: File
path: hello.js
```
```
/data/test$ cat hello.js
console.log("Hello World");
```

## Full Traceback
```
/data/test$ ls
docker-job.yml docker.cwl hello.js tests
```

```pytb
/data/test$ cwl-runner --record-container-id --cidfile-dir=./tests --debug docker.cwl docker-job.yml
/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/bin/cwl-runner 1.0.20180820141117
Resolved 'docker.cwl' to 'file:///data/test/docker.cwl'
[job docker.cwl] initializing from file:///data/test/docker.cwl
[job docker.cwl] {
"src": {
"class": "File",
"location": "file:///data/test/hello.js",
"size": 28,
"basename": "hello.js",
"nameroot": "hello",
"nameext": ".js"
}
}
[job docker.cwl] path mappings is {
"file:///data/test/hello.js": [
"/data/test/hello.js",
"/var/lib/cwl/stgb8052744-f132-4e0f-8c6a-85162259489f/hello.js",
"File",
true
]
}
[job docker.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "node"
},
{
"position": [
1,
"src"
],
"datum": {
"class": "File",
"location": "file:///data/test/hello.js",
"size": 28,
"basename": "hello.js",
"nameroot": "hello",
"nameext": ".js",
"path": "/var/lib/cwl/stgb8052744-f132-4e0f-8c6a-85162259489f/hello.js",
"dirname": "/var/lib/cwl/stgb8052744-f132-4e0f-8c6a-85162259489f"
}
}
]
[job docker.cwl] /tmp/tmp0l81p3bv$ docker \
run \
-i \
--volume=/tmp/tmp0l81p3bv:/var/spool/cwl:rw \
--volume=/tmp/tmpzf79qfo0:/tmp:rw \
--volume=/data/test/hello.js:/var/lib/cwl/stgb8052744-f132-4e0f-8c6a-85162259489f/hello.js:ro \
--workdir=/var/spool/cwl \
--read-only=true \
--log-driver=none \
--user=201808:700 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/var/spool/cwl \
--cidfile=./tests/20190213123723-316187.cid \
node:slim \
node \
/var/lib/cwl/stgb8052744-f132-4e0f-8c6a-85162259489f/hello.js > /tmp/tmp0l81p3bv/output.txt
docker: Failed to create the container ID file: open ./tests/20190213123723-316187.cid: no such file or directory.
See 'docker run --help'.
[job docker.cwl] completed permanentFail
[job docker.cwl] {
"example_out": {
"location": "file:///tmp/tmp0l81p3bv/output.txt",
"basename": "output.txt",
"nameroot": "output",
"nameext": ".txt",
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0,
"http://commonwl.org/cwltool#generation": 0
}
}
[job docker.cwl] Removing input staging directory /tmp/tmpfn0ztuxb
[job docker.cwl] Removing temporary directory /tmp/tmpzf79qfo0
Moving /tmp/tmp0l81p3bv/output.txt to /data/test/output.txt
Removing intermediate output directory /tmp/tmp0l81p3bv
{
"example_out": {
"location": "file:///data/test/output.txt",
"basename": "output.txt",
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0,
"path": "/data/test/output.txt"
}
}
Final process status is permanentFail
```

Successful run with absolute path

```
/data/test$ cwl-runner --record-container-id --cidfile-dir=/data/test/tests --debug docker.cwl docker-job.yml
/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/bin/cwl-runner 1.0.20180820141117
Resolved 'docker.cwl' to 'file:///data/test/docker.cwl'
[job docker.cwl] initializing from file:///data/test/docker.cwl
[job docker.cwl] {
"src": {
"class": "File",
"location": "file:///data/test/hello.js",
"size": 28,
"basename": "hello.js",
"nameroot": "hello",
"nameext": ".js"
}
}
[job docker.cwl] path mappings is {
"file:///data/test/hello.js": [
"/data/test/hello.js",
"/var/lib/cwl/stge01751cb-1a70-4ea1-a606-aee7de4f43c5/hello.js",
"File",
true
]
}
[job docker.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "node"
},
{
"position": [
1,
"src"
],
"datum": {
"class": "File",
"location": "file:///data/test/hello.js",
"size": 28,
"basename": "hello.js",
"nameroot": "hello",
"nameext": ".js",
"path": "/var/lib/cwl/stge01751cb-1a70-4ea1-a606-aee7de4f43c5/hello.js",
"dirname": "/var/lib/cwl/stge01751cb-1a70-4ea1-a606-aee7de4f43c5"
}
}
]
[job docker.cwl] /tmp/tmp5z1gpfeu$ docker \
run \
-i \
--volume=/tmp/tmp5z1gpfeu:/var/spool/cwl:rw \
--volume=/tmp/tmp04upfri_:/tmp:rw \
--volume=/data/test/hello.js:/var/lib/cwl/stge01751cb-1a70-4ea1-a606-aee7de4f43c5/hello.js:ro \
--workdir=/var/spool/cwl \
--read-only=true \
--log-driver=none \
--user=201808:700 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/var/spool/cwl \
--cidfile=/data/test/tests/20190213123830-261962.cid \
node:slim \
node \
/var/lib/cwl/stge01751cb-1a70-4ea1-a606-aee7de4f43c5/hello.js > /tmp/tmp5z1gpfeu/output.txt
[job docker.cwl] completed success
[job docker.cwl] {
"example_out": {
"location": "file:///tmp/tmp5z1gpfeu/output.txt",
"basename": "output.txt",
"nameroot": "output",
"nameext": ".txt",
"class": "File",
"checksum": "sha1$648a6a6ffffdaa0badb23b8baf90b6168dd16b3a",
"size": 12,
"http://commonwl.org/cwltool#generation": 0
}
}
[job docker.cwl] Removing input staging directory /tmp/tmp3u_vaonu
[job docker.cwl] Removing temporary directory /tmp/tmp04upfri_
Moving /tmp/tmp5z1gpfeu/output.txt to /data/test/output.txt
Removing intermediate output directory /tmp/tmp5z1gpfeu
{
"example_out": {
"location": "file:///data/test/output.txt",
"basename": "output.txt",
"class": "File",
"checksum": "sha1$648a6a6ffffdaa0badb23b8baf90b6168dd16b3a",
"size": 12,
"path": "/data/test/output.txt"
}
}
Final process status is success
```

## Your Environment
* cwltool version: 1.0.20180820141117
* docker version: Docker version 18.09.1, build 4c52b90

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.