common-workflow-language / common-workflow-language/cwltool
When using Singularity, files cannot be mounted properly to the initial directory.
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
Proper mounting of files on containers.
## Actual Behavior
When I executed `cwl-runner --singularity --debug PATH_TO_CWL_FILE PATH_TO_YAML_FILE`, I sometimes got an error as above.
```
...
INFO PATH_TO_CWL_RUNNER_BIN/bin/cwl-runner 1.0.20190815141648
INFO Resolved 'PATH_TO_CWL_FILE' to 'file://PATH_TO_CWL_FILE'
../../../..PATH_TO_CWL_FILE:41:5: object id `../../../..PATH_TO_CWL_FILE#log_file` previously defined
DEBUG [job CWL_FILE] initializing from file://PATH_TO_CWL_FILE
INFO Using local copy of Singularity image found in /yshare1/home/suzuki/test_dir
DEBUG [job CWL_FILE] initial work dir {
"file:///yshare1/home/suzuki/test_dir/FILE_NAME": [
"/yshare1/home/suzuki/test_dir/FILE_NAME",
"/BBWRqe/ENTRY_DIR_NAME/FILE_NAME",
"WritableFile",
true
],
"_:e69c7b49-688e-49ca-8d78-c47d4ae6fcee": [
"_:e69c7b49-688e-49ca-8d78-c47d4ae6fcee",
"/BBWRqe/ENTRY_DIR_NAME",
"WritableDirectory",
true
]
}
ERROR Got workflow error
Traceback (most recent call last):
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/executors.py", line 188, in run_jobs
job.run(runtime_context)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/job.py", line 674, in run
(runtime, cidfile) = self.create_runtime(env, runtimeContext)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/singularity.py", line 355, in create_runtime
tmpdir_prefix=runtime_context.tmpdir_prefix)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/job.py", line 590, in add_volumes
runtime, vol, host_outdir_tgt, tmpdir_prefix)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/singularity.py", line 266, in add_writable_file_volume
shutil.copy(volume.resolved, host_outdir_tgt)
File "/usr/local/package/python/2.7.15/lib/python2.7/shutil.py", line 133, in copy
copyfile(src, dst)
File "/usr/local/package/python/2.7.15/lib/python2.7/shutil.py", line 97, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: u'/tmp/SgOUR3/ENTRY_DIR_NAME/FILE_NAME'
ERROR Workflow error:
[Errno 2] No such file or directory: u'/tmp/SgOUR3/ENTRY_DIR_NAME/FILE_NAME'
Traceback (most recent call last):
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/main.py", line 830, in main
logger=_logger)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/executors.py", line 46, in __call__
return self.execute(*args, **kwargs)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/executors.py", line 106, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/cwltool/executors.py", line 196, in run_jobs
raise_from(WorkflowException(Text(err)), err)
File "PATH_TO_CWL_RUNNER_BIN/lib/python2.7/site-packages/future/utils/__init__.py", line 456, in raise_from
raise e
WorkflowException: [Errno 2] No such file or directory: u'/tmp/SgOUR3/ENTRY_DIR_NAME/FILE_NAME'
```
But sometimes it can be successful. The difference between the two debug messages is the order in the InitialWorkdir part.
```
...
DEBUG [job CWL_FILE] initial work dir {
"_:4bd661c4-633e-4345-9495-ddb7cb5be4ff": [
"_:4bd661c4-633e-4345-9495-ddb7cb5be4ff",
"/oaFYMn/ENTRY_DIR_NAME",
"WritableDirectory",
true
],
"file:///yshare1/home/suzuki/test_dir/FILE_NAME": [
"/yshare1/home/suzuki/test_dir/FILE_NAME",
"/oaFYMn/ENTRY_DIR_NAME/FILE_NAME",
"WritableFile",
true
]
}
...
INFO Final process status is success
```
As this error does not occur in Docker, there may be a bug in the Singularity related code. It may be solved by controlling so that Workdir comes first in the order of files when using InitialWorkdir.
## Workflow Code
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
label: CWL script to run denoizing by deblur
baseCommand: ["XXX", "YYY", "ZZZ"]
requirements:
InlineJavascriptRequirement: {}
SchemaDefRequirement:
types:
- $import: XXX.yaml
InitialWorkDirRequirement:
listing:
- entryname: ENTRY_DIR_NAME
entry: "$({class: 'Directory', listing: inputs.seqs_file_pathes})"
writable: true
hints:
DockerRequirement:
dockerPull: XXX...
inputs:
# input file
seqs_file_pathes:
type: File[]
...
```
## YAML Code
```
seqs_file_pathes:
- class: File
path: PATH_TO_FILE
```
## Your Environment
* cwltool version: 1.0.20190815141648
* Python: 2.7.15
* Singularity: 3.2.1
* Node.js: 12.3.1
Contributor guide
Assessment
This issue has not been assessed yet.