common-workflow-language / common-workflow-language/cwltool

Problem with Directory output

Offen
#361 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
376
Forks
255
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
12

Beschreibung

### New version
**1.0.20170329142446 (other requirements in attached file)**

I use Directory output in my pipiline, and I recently stumbled upon an **Errno21 :**
(*My current pipeline runs on old version of cwltool and cwlfile resembles last example, the tool used in my pipeline creates directory and then placed files into it*)

**Output:**
`[Errno 21] Is a directory: '/mnt/projects/.../cwl_bug/cwl_bug_clt_cache_new_cwltool/a6f81c57e16b291e6dea2e85b9a91122/123_dir'`
when cwl tries to copy result directory from cache to outdir.

**Libraries' versions:** [new_requirements.txt](https://github.com/common-workflow-language/cwltool/files/912423/new_requirements.txt)

**Command:**
`cwltool --outdir cwl_bug_clt_new_cwltool --cache cwl_bug_clt_cache_new_cwltool cwl_bug_clt.cwl`

**Cwl file:**
```
cwlVersion: v1.0
class: CommandLineTool

baseCommand: [mkdir, 123_dir]

inputs: []
outputs:
dir:
type: Directory
outputBinding:
glob: '123_dir'
```

**Debug output:**
```
[job cwl_bug_clt.cwl] /mnt/projects/.../cwl_bug/cwl_bug_clt_cache_new_cwltool/e25d330df2dbcbaf7614379c587fd113$ mkdir \
123_dir
[job cwl_bug_clt.cwl] completed success
[job cwl_bug_clt.cwl] {
"dir": {
"basename": "123_dir",
"location": "file:///mnt/projects/.../cwl_bug/cwl_bug_clt_cache_new_cwltool/e25d330df2dbcbaf7614379c587fd113/123_dir",
"class": "Directory"
}
}
[job cwl_bug_clt.cwl] Removing input staging directory /tmp/tmp6DGGcP
[job cwl_bug_clt.cwl] Removing temporary directory /tmp/tmp3c3Vk4
Copying /mnt/.../cwl_bug/cwl_bug_clt_cache_new_cwltool/e25d330df2dbcbaf7614379c587fd113/123_dir to cwl_bug_clt_new_cwltool/123_dir
Unhandled error, try again with --debug for more information:
[Errno 21] Is a directory: '/mnt/projects/.../cwl_bug/cwl_bug_clt_cache_new_cwltool/e25d330df2dbcbaf7614379c587fd113/123_dir'
Traceback (most recent call last):
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/main.py", line 768, in main
**vars(args))
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/main.py", line 252, in single_job_executor
kwargs["make_fs_access"](""))
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/process.py", line 242, in relocateOutputs
stageFiles(pm, moveIt)
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/process.py", line 193, in stageFiles
stageFunc(p.resolved, p.target)
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/process.py", line 237, in moveIt
shutil.copy(src, dst)
File "/usr/lib/python2.7/shutil.py", line 119, in copy
copyfile(src, dst)
File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 21] Is a directory: '/mnt/projects/.../cwl_bug/cwl_bug_clt_cache_new_cwltool/e25d330df2dbcbaf7614379c587fd113/123_dir'
```

Job is completed successfully and cwl fails only on copy from cache.

If runned without --cache, it raises **Errno 2:**
**Debug output:**
```
[job cwl_bug_clt.cwl] completed success
[job cwl_bug_clt.cwl] {
"dir": {
"basename": "123_dir",
"location": "file:///tmp/tmpHs9ukN/123_dir",
"class": "Directory"
}
}
[job cwl_bug_clt.cwl] Removing input staging directory /tmp/tmpZg4naI
[job cwl_bug_clt.cwl] Removing temporary directory /tmp/tmprqKLfQ
[job cwl_bug_clt.cwl] Removing empty output directory /tmp/tmpHs9ukN
Unhandled error, try again with --debug for more information:
[Errno 2] No such file or directory: '/tmp/tmpHs9ukN/123_dir'
Traceback (most recent call last):
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/main.py", line 768, in main
**vars(args))
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/main.py", line 252, in single_job_executor
kwargs["make_fs_access"](""))
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/process.py", line 224, in relocateOutputs
adjustDirObjs(outputObj, functools.partial(get_listing, fs_access, recursive=True))
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 55, in adjustDirObjs
adjustDirObjs(rec[key], op)
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 53, in adjustDirObjs
op(rec)
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 111, in get_listing
for ld in fs_access.listdir(loc):
File "/home/aslepchenkov/cwl_venv_latest/local/lib/python2.7/site-packages/cwltool/stdfsaccess.py", line 38, in listdir
return [abspath(urllib.quote(str(l)), fn) for l in os.listdir(self._abs(fn))]
OSError: [Errno 2] No such file or directory: '/tmp/tmpHs9ukN/123_dir'
```

### Old version
**1.0.20170119234115 (other requirements in attached file)**

And there are more strange things:

When runned with **older version: **[old_requirements.txt](https://github.com/common-workflow-language/cwltool/files/912425/old_requirements.txt).

No exceptions are raised but no directory is created either. However output log states that it is created:

**Output:**
```
Resolved 'cwl_bug_clt.cwl' to 'file:///mnt/projects/.../cwl_bug/cwl_bug_clt.cwl'
[job cwl_bug_clt.cwl] Output of job will be cached in /mnt/projects/.../cwl_bug/cwl_bug_clt_cache/e25d330df2dbcbaf7614379c587fd113
[job cwl_bug_clt.cwl] /mnt/projects/.../cwl_bug/cwl_bug_clt_cache/e25d330df2dbcbaf7614379c587fd113$ mkdir \
123_dir
[job cwl_bug_clt.cwl] completed success
{
"dir": {
"path": "/mnt/projects/.../cwl_bug/123_dir",
"basename": "123_dir",
"listing": [],
"location": "file:///mnt/projects/.../cwl_bug/123_dir",
"class": "Directory"
}
}
Final process status is success
```

No directory here:
```
(pipeline_venv) aslepchenkov@bioserver:/mnt/projects/.../cwl_bug$ ls
cwl_bug_clt cwl_bug_clt_cache cwl_bug_clt_cache_new_cwltool cwl_bug_clt.cwl cwl_bug_clt_new_cwltool cwl_bug_wf.cwl new_requirements.txt old_requirements.txt pgm
```

The situation is changed when I create something inside this directory:

**Cwl file:**
```
cwlVersion: v1.0
class: CommandLineTool

baseCommand: [mkdir, 123_dir, "&&", touch, 123_dir/123.txt]

inputs: []
outputs:
dir:
type: Directory
outputBinding:
glob: '123_dir'
```

Now directory is created, but there is no file inside it. **However output says that soomething should be there and it's of directory class:**

**Output:**
```
[job cwl_bug_clt.cwl] /mnt/projects/.../cwl_bug/cwl_bug_clt_cache/a6f81c57e16b291e6dea2e85b9a91122$ mkdir \
123_dir \
&& \
touch \
123_dir/123.txt
[job cwl_bug_clt.cwl] completed success
{
"dir": {
"path": "/mnt/projects/.../cwl_bug/123_dir",
"basename": "123_dir",
"listing": [
{
"path": "/mnt/projects/.../cwl_bug/123_dir/123.txt",
"basename": "123.txt",
"listing": [],
"class": "Directory",
"location": "file:///mnt/projects/.../cwl_bug/123_dir/123.txt"
}
],
"location": "file:///mnt/projects/.../cwl_bug/123_dir",
"class": "Directory"
}
}
Final process status is success
```

```
(pipeline_venv) aslepchenkov@bioserver:/mnt/projects/.../cwl_bug$ ls
123_dir cwl_bug_clt cwl_bug_clt_cache cwl_bug_clt_cache_new_cwltool cwl_bug_clt.cwl cwl_bug_clt_new_cwltool cwl_bug_wf.cwl new_requirements.txt old_requirements.txt pgm

(pipeline_venv) aslepchenkov@bioserver:/mnt/projects/.../cwl_bug$ ll 123_dir
total 25
drwxr-xr-x 2 aslepchenkov ... 2 Apr 11 10:41 ./
drwxrwx--- 8 aslepchenkov ... 12 Apr 11 10:41 ../
```

And finally when I do something like this:

**Cwl file:**
```
cwlVersion: v1.0
class: CommandLineTool

requirements:
- class: ShellCommandRequirement

baseCommand: [mkdir, 123_dir]

inputs: []

arguments:
- shellQuote: False
- valueFrom: "&&"
position: 1
- valueFrom: "touch"
position: 2
- valueFrom: "123_dir/123.txt"
position: 3

outputs:
file:
type: Directory
outputBinding:
glob: '123_dir'
```

Directory and file inside it are prorperly created and placed into output directory.

**I agree that my second cwl file may use base command the wrong way, and created 123.txt file is of Directory class by this reason. However behaviour of the new version of cwltool and old version with blank directory seems strange to me.**

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.