Docker: directory mount fails

Tested for cwltool 1.0.20170327143622
Docker is trying to mount not only File but Directory too. When Directory is returned from expression tool it fails to mount.
Testing example: https://github.com/michael-kotliar/cwl_sandbox/tree/master/docker_mount
To run: `cwltool --debug workflow.cwl job.yml`

Detailed log

  

   Resolved 'workflow.cwl' to 'file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl'

[workflow workflow.cwl] initialized from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl

[workflow workflow.cwl] workflow starting

[workflow workflow.cwl] starting step files_to_folder

[job step files_to_folder] job input {

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#files_to_folder/input_files": [

        {

            "class": "File", 

            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt", 

            "basename": "empty_file_1.txt"

        }, 

        {

            "class": "File", 

            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt", 

            "basename": "empty_file_2.txt"

        }

    ]

}

[job step files_to_folder] evaluated job input to {

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#files_to_folder/input_files": [

        {

            "class": "File", 

            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt", 

            "basename": "empty_file_1.txt"

        }, 

        {

            "class": "File", 

            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt", 

            "basename": "empty_file_2.txt"

        }

    ]

}

[step files_to_folder] produced output {

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#files_to_folder/folder": {

        "basename": "folder", 

        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 

        "class": "Directory", 

        "listing": [

            {

                "basename": "empty_file_1.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"

            }, 

            {

                "basename": "empty_file_2.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"

            }

        ]

    }

}

[step files_to_folder] completed success

[workflow workflow.cwl] starting step list_dir

[job step list_dir] job input {

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#list_dir/input_dir": {

        "basename": "folder", 

        "listing": [

            {

                "basename": "empty_file_1.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"

            }, 

            {

                "basename": "empty_file_2.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"

            }

        ], 

        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 

        "class": "Directory"

    }

}

[job step list_dir] evaluated job input to {

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#list_dir/input_dir": {

        "basename": "folder", 

        "listing": [

            {

                "basename": "empty_file_1.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"

            }, 

            {

                "basename": "empty_file_2.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"

            }

        ], 

        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 

        "class": "Directory"

    }

}

[job list_dir] initializing from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/list_dir.cwl as part of step list_dir

[job list_dir] {

    "input_dir": {

        "basename": "folder", 

        "listing": [

            {

                "basename": "empty_file_1.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"

            }, 

            {

                "basename": "empty_file_2.txt", 

                "class": "File", 

                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"

            }

        ], 

        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 

        "class": "Directory"

    }

}

[job list_dir] path mappings is {

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt": [

        "/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt", 

        "/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_1.txt", 

        "File", 

        true

    ], 

    "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c": [

        "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 

        "/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder", 

        "Directory", 

        true

    ], 

    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt": [

        "/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt", 

        "/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_2.txt", 

        "File", 

        true

    ]

}

[job list_dir] command line bindings is [

    {

        "position": [

            -1000000, 

            0

        ], 

        "datum": "ls"

    }, 

    {

        "valueFrom": "$(inputs.input_dir)", 

        "position": [

            1, 

            0

        ]

    }

]

[job list_dir] /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmphPh_f3$ docker \

    run \

    -i \

    --volume=/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_1.txt:ro \

    --volume=/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_2.txt:ro \

    --volume=_:472fab15-63fb-42fd-9c0a-1fc99830cd2c:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder:ro \ 

    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmphPh_f3:/private/var/spool/cwl:rw \

    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpLyN4Rb:/tmp:rw \

    --workdir=/private/var/spool/cwl \

    --read-only=true \

    --user=501 \

    --rm \

    --env=TMPDIR=/tmp \

    --env=HOME=/private/var/spool/cwl \

    ubuntu \

    ls \

    /private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder

docker: Error response from daemon: invalid bind mount spec "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder:ro": invalid volume specification: '_:472fab15-63fb-42fd-9c0a-1fc99830cd2c:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder:ro'.

See 'docker run --help'.

[job list_dir] completed permanentFail

[job list_dir] {}

[step list_dir] produced output {}

[step list_dir] completed permanentFail

[job list_dir] Removing input staging directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpFbJ2B9

[job list_dir] Removing temporary directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpLyN4Rb

[job list_dir] Removing empty output directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmphPh_f3

[workflow workflow.cwl] outdir is /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpdX4XQf

Removing intermediate output directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpdX4XQf

{}

Final process status is permanentFail


   

  

Start by running `cwltool --debug workflow.cwl job.yml` with the linked docker_mount example and compare the generated Docker volume mappings with the detailed log. Read `workflow.cwl` and `list_dir.cwl`; done means a Directory returned from an expression mounts successfully and the `list_dir` step completes instead of producing an invalid bind mount error.

Written by the indexing model from the issue text.