common-workflow-language / common-workflow-language/cwltool

[question] selecting right parameter for 2 step mapping of 'output' from docker

Open
#1,295 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I would like some guidance, as I'm not to sure which parameters to employ/set for running some operation (see detailed explanation after). Basically, I see all following parameters are available, but I don't know which one(s) to pick to accomplish desired behavior, mostly because I'm confused by how they interact with one-another.

`RuntimeContext`
```
tmpdir_prefix
tmp_outdir_prefix
basedir
outdir
tmpdir
stagedir
docker_outdir
docker_tmpdir
docker_stagedir
path_mapper
```

`JobBase`
```
pathmapper
make_path_mapper
generatemapper
```

# Explanation

I have a setup with a "parent" docker that runs cwltool inside it, and it needs to run another "child" docker application. My docker-compose is configured to have sibling containers. I have tested this by manually running cwltool from command line within the "parent" docker using `--tmpdir-prefix` and `--tmp-outdir-prefix` arguments pointing to mounted volumes and it works *only* because I mount my volume with the same path (e.g.: `/tmpdir/inputs:/tmpdir/inputs`). This allows files to be found during both CWL I/O location resolution (viewed from within "parent" docker) and during the actual docker execution (viewed from outside "parent" docker / docker-compose side).

My problem is that now I need to replicate the same thing directly with cwltool's `RuntimeContext`, *and* I want to place the files at a specific location that doesn't have the same path (eg: `/custom/outputs:/tmpdir/outputs`). Since the "child" docker is running as sibling to the "parent", it does not see the final output location inside the "parent" (`/tmpdir/outputs`) as it runs side-by-side with it's own volumes. It can only see `/custom/outputs`. So I need a way to tell cwltool some location mapping such that "child" docker app can solve as if pointing to `/tmpdir/outputs` of the "parent".

**Edit: extra detail**

cwltool will see inside "parent" dirs mounted as such during I/O resolution
```
/tmp/server-side/inputs => /tmp/parent-side/inputs
/tmp/server-side/outputs => /tmp/parent-side/outputs
```
but cwltool-docker-job needs to resolve the "corresponding" paths as:
```
docker run -v /tmp/server-side/inputs:/tmp/whatever -v /tmp/server-side/outputs:/tmp/randout
```
because it is executed outside the "parent" docker where I/O resolution was done.

Which (combination of) parameter(s) should be more appropriate to achieve this goal?
Thanks a lot for your help.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.