common-workflow-language / common-workflow-language/cwltool

cached cwl.output.json contains docker file path references

Open
#1,573 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

Hi all,

I am experimenting with `cwl.output.json` to get the results back from a CommandLineTool that executes in a Docker environment. This works fine, but there appears to be a problem when re-running the same workflow: `cwltool` correctly recognizes that the cache can be used, but it chokes on the filepath that was saved into the `cwl.output.json` file which contains a path to a random generated folder that was used during the first run.

## Expected Behavior
Caching should work fine, as expected

## Actual Behavior
When retrying, I get the following error:

```
cwltool --outdir output --cachedir cache spike.cwl spike.yaml
INFO /usr/local/miniforge3/bin/cwltool 3.1.20211107152837
INFO Resolved 'spike.cwl' to 'file:///home/tom/gitlab/cwl-workflows/workflows/spike.cwl'
spike.cwl:8:3: Warning: checking item
Warning: Field `class` contains undefined reference to
`http://commonwl.org/cwltool#Secrets`
INFO spike.cwl:8:3: Unknown hint http://commonwl.org/cwltool#Secrets
INFO [workflow ] start
INFO [workflow ] starting step arv_get
INFO [step arv_get] start
INFO [job arv_get] Using cached output in /home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511
ERROR Workflow error, try again with --debug for more information:
Output file path /cByOZc/ubuntu.sif must be within designated output directory (/nnKPqR) or an input file pass through.
```
The initial run produced:

```
INFO /usr/local/miniforge3/bin/cwltool 3.1.20211107152837
INFO Resolved 'spike.cwl' to 'file:///home/tom/gitlab/cwl-workflows/workflows/spike.cwl'
spike.cwl:8:3: Warning: checking item
Warning: Field `class` contains undefined reference to
`http://commonwl.org/cwltool#Secrets`
INFO spike.cwl:8:3: Unknown hint http://commonwl.org/cwltool#Secrets
INFO [workflow ] start
INFO [workflow ] starting step arv_get
INFO [step arv_get] start
INFO [job arv_get] Output of job will be cached in /home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511
INFO [job arv_get] /home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511$ docker \
run \
-i \
--mount=type=bind,source=/home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511,target=/cByOZc \
--mount=type=bind,source=/tmp/0n3iy0j2,target=/tmp \
--workdir=/cByOZc \
--read-only=true \
--user=1002:1002 \
--rm \
--cidfile=/tmp/z5bv8n5s/20211207145621-197335.cid \
--env=TMPDIR=/tmp \
--env=HOME=/cByOZc \
arv-cli:build-tar-fd145ede211e86f23f7aeab39e45de43 \
arv-get-cwl
INFO [job arv_get] Max memory used: 47MiB
INFO [job arv_get] completed success
INFO [step arv_get] completed success
INFO [workflow ] completed success
{
"collection_file": [
{
"class": "File",
"basename": "ubuntu.sif",
"location": "file:///home/tom/gitlab/cwl-workflows/workflows/output/ubuntu.sif",
"checksum": "sha1$8a13313f5de5ace0d943ff7a3257fc83c0538829",
"size": 27742208,
"path": "/home/tom/gitlab/cwl-workflows/workflows/output/ubuntu.sif"
}
]
}
INFO Final process status is success
```
## Workflow Code

CommandLineTool `arv-get.cwl`:

```yaml
cwlVersion: v1.2
class: CommandLineTool

requirements:
DockerRequirement:
dockerPull: arv-cli
NetworkAccess:
networkAccess: true
InitialWorkDirRequirement:
listing:
- entryname: cwl.inputs.json
entry: '{"inputs": $(inputs)"}'

baseCommand:
- arv-get-cwl

inputs:
arvados_collection_locator: string
arvados_api_token: string
arvados_api_host: string

outputs:
collection_file: File
```

The `arv-get-cwl` script within the container extracts the input from `cwl.inputs.json` and passes it to the `arv-get` command, after which the `cwl.output.json` file is produced with the filename:

```bash
cat > ${outputfile} <

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.