common-workflow-language / common-workflow-language/cwltool

Inputs containing both directories and files within directories fail to get mounted to correct places for CWL >= 1.1 workflows

Open
#1,789 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

If you pass in a directory and a file contained within this directory to a CWL tool using cwl>=1.1, the file contained
within this directory does not get put into the correct location for the tool to run.

## Expected Behavior
The file contained within the directory should get put in the correct location.

## Actual Behavior
The file contained within the directory does not get put in the correct location when you use cwl >= 1.1.

## Workflow Code
`tool.cwl`
```
cwlVersion: v1.1
class: CommandLineTool
id: tool

baseCommand:
- ls

inputs:
file:
type: File
inputBinding:
position: 1
directory:
type: Directory
inputBinding:
position: 2

outputs: []

```

`inputs.yaml`
```
directory:
class: Directory
path: directory
file:
class: File
path: directory/file

```

## Full Traceback
```pytb
$ cwltool --debug tool.cwl inputs.yaml
INFO /home/user/Documents/checkouts/cwltool/venv/bin/cwltool 3.1.20221115160822
INFO Resolved 'tool.cwl' to 'file:///home/user/Documents/checkouts/cwltool/my/issue2/tool.cwl'
DEBUG [job tool] initializing from file:///home/user/Documents/checkouts/cwltool/my/issue2/tool.cwl#tool
DEBUG [job tool] {
"directory": {
"class": "Directory",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory",
"basename": "directory"
},
"file": {
"class": "File",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"size": 0,
"basename": "file",
"nameroot": "file",
"nameext": ""
}
}
DEBUG [job tool] path mappings is {
"file:///home/user/Documents/checkouts/cwltool/my/issue2/directory": [
"/home/user/Documents/checkouts/cwltool/my/issue2/directory",
"/tmp/qu6q87wp/stgdcb1db8c-5f78-4354-b36e-915ec91ebab8/directory",
"Directory",
true
],
"file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file": [
"/home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"/tmp/qu6q87wp/stg0fec546f-112c-4c2c-afa0-17256f9bd17a/file",
"File",
true
]
}
DEBUG [job tool] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "ls"
},
{
"position": [
1,
"file"
],
"datum": {
"class": "File",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"size": 0,
"basename": "file",
"nameroot": "file",
"nameext": "",
"path": "/tmp/qu6q87wp/stg0fec546f-112c-4c2c-afa0-17256f9bd17a/file",
"dirname": "/tmp/qu6q87wp/stg0fec546f-112c-4c2c-afa0-17256f9bd17a"
}
},
{
"position": [
2,
"directory"
],
"datum": {
"class": "Directory",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory",
"basename": "directory",
"path": "/tmp/qu6q87wp/stgdcb1db8c-5f78-4354-b36e-915ec91ebab8/directory",
"dirname": "/tmp/qu6q87wp/stgdcb1db8c-5f78-4354-b36e-915ec91ebab8"
}
}
]
DEBUG [job tool] initial work dir {}
INFO [job tool] /tmp/89n9yj5o$ ls \
/tmp/qu6q87wp/stg0fec546f-112c-4c2c-afa0-17256f9bd17a/file \
/tmp/qu6q87wp/stgdcb1db8c-5f78-4354-b36e-915ec91ebab8/directory
ls: cannot access '/tmp/qu6q87wp/stg0fec546f-112c-4c2c-afa0-17256f9bd17a/file': No such file or directory
/tmp/qu6q87wp/stgdcb1db8c-5f78-4354-b36e-915ec91ebab8/directory:
file
DEBUG Could not collect memory usage, job ended before monitoring began.
WARNING [job tool] exited with status: 2
WARNING [job tool] completed permanentFail
DEBUG [job tool] outputs {}
DEBUG [job tool] Removing input staging directory /tmp/qu6q87wp
DEBUG [job tool] Removing temporary directory /tmp/pk_1qhky
DEBUG Removing intermediate output directory /tmp/89n9yj5o
{}WARNING Final process status is permanentFail
```

If we use CWL v1.0, this runs successfully:
```
$ cwltool --debug tool.cwl inputs.yaml
INFO /home/user/Documents/checkouts/cwltool/venv/bin/cwltool 3.1.20221115160822
INFO Resolved 'tool.cwl' to 'file:///home/user/Documents/checkouts/cwltool/my/issue2/tool.cwl'
DEBUG [job tool] initializing from file:///home/user/Documents/checkouts/cwltool/my/issue2/tool.cwl#tool
DEBUG [job tool] {
"directory": {
"class": "Directory",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory",
"basename": "directory",
"listing": [
{
"class": "File",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"basename": "file",
"size": 0
}
]
},
"file": {
"class": "File",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"size": 0,
"basename": "file",
"nameroot": "file",
"nameext": ""
}
}
DEBUG [job tool] path mappings is {
"file:///home/user/Documents/checkouts/cwltool/my/issue2/directory": [
"/home/user/Documents/checkouts/cwltool/my/issue2/directory",
"/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory",
"Directory",
true
],
"file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file": [
"/home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory/file",
"File",
false
]
}
DEBUG [job tool] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "ls"
},
{
"position": [
1,
"file"
],
"datum": {
"class": "File",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"size": 0,
"basename": "file",
"nameroot": "file",
"nameext": "",
"path": "/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory/file",
"dirname": "/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory"
}
},
{
"position": [
2,
"directory"
],
"datum": {
"class": "Directory",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory",
"basename": "directory",
"listing": [
{
"class": "File",
"location": "file:///home/user/Documents/checkouts/cwltool/my/issue2/directory/file",
"basename": "file",
"size": 0,
"path": "/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory/file",
"dirname": "/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory",
"nameroot": "file",
"nameext": ""
}
],
"path": "/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory",
"dirname": "/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208"
}
}
]
DEBUG [job tool] initial work dir {}
INFO [job tool] /tmp/0aycvk8u$ ls \
/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory/file \
/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory
/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory/file

/tmp/4k4zjduc/stg81d0fed8-0aa8-498c-b9c2-54a06ad10208/directory:
file
DEBUG Could not collect memory usage, job ended before monitoring began.
INFO [job tool] completed success
DEBUG [job tool] outputs {}
DEBUG [job tool] Removing input staging directory /tmp/4k4zjduc
DEBUG [job tool] Removing temporary directory /tmp/mt4vd3cg
DEBUG Removing intermediate output directory /tmp/0aycvk8u
{}INFO Final process status is success
```

## Your Environment
* cwltool version:
```
$ cwltool --version
/home/user/Documents/checkouts/cwltool/venv/bin/cwltool 3.1.20221115160822
```

(using the tip of `main`)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.