common-workflow-language / common-workflow-language/cwltool
cached cwl.output.json contains docker file path references
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
Hi all,
I am experimenting with `cwl.output.json` to get the results back from a CommandLineTool that executes in a Docker environment. This works fine, but there appears to be a problem when re-running the same workflow: `cwltool` correctly recognizes that the cache can be used, but it chokes on the filepath that was saved into the `cwl.output.json` file which contains a path to a random generated folder that was used during the first run.
## Expected Behavior
Caching should work fine, as expected
## Actual Behavior
When retrying, I get the following error:
```
cwltool --outdir output --cachedir cache spike.cwl spike.yaml
INFO /usr/local/miniforge3/bin/cwltool 3.1.20211107152837
INFO Resolved 'spike.cwl' to 'file:///home/tom/gitlab/cwl-workflows/workflows/spike.cwl'
spike.cwl:8:3: Warning: checking item
Warning: Field `class` contains undefined reference to
`http://commonwl.org/cwltool#Secrets`
INFO spike.cwl:8:3: Unknown hint http://commonwl.org/cwltool#Secrets
INFO [workflow ] start
INFO [workflow ] starting step arv_get
INFO [step arv_get] start
INFO [job arv_get] Using cached output in /home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511
ERROR Workflow error, try again with --debug for more information:
Output file path /cByOZc/ubuntu.sif must be within designated output directory (/nnKPqR) or an input file pass through.
```
The initial run produced:
```
INFO /usr/local/miniforge3/bin/cwltool 3.1.20211107152837
INFO Resolved 'spike.cwl' to 'file:///home/tom/gitlab/cwl-workflows/workflows/spike.cwl'
spike.cwl:8:3: Warning: checking item
Warning: Field `class` contains undefined reference to
`http://commonwl.org/cwltool#Secrets`
INFO spike.cwl:8:3: Unknown hint http://commonwl.org/cwltool#Secrets
INFO [workflow ] start
INFO [workflow ] starting step arv_get
INFO [step arv_get] start
INFO [job arv_get] Output of job will be cached in /home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511
INFO [job arv_get] /home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511$ docker \
run \
-i \
--mount=type=bind,source=/home/tom/gitlab/cwl-workflows/workflows/cache/d04184a5b32119f4058d7e8fbc6ff511,target=/cByOZc \
--mount=type=bind,source=/tmp/0n3iy0j2,target=/tmp \
--workdir=/cByOZc \
--read-only=true \
--user=1002:1002 \
--rm \
--cidfile=/tmp/z5bv8n5s/20211207145621-197335.cid \
--env=TMPDIR=/tmp \
--env=HOME=/cByOZc \
arv-cli:build-tar-fd145ede211e86f23f7aeab39e45de43 \
arv-get-cwl
INFO [job arv_get] Max memory used: 47MiB
INFO [job arv_get] completed success
INFO [step arv_get] completed success
INFO [workflow ] completed success
{
"collection_file": [
{
"class": "File",
"basename": "ubuntu.sif",
"location": "file:///home/tom/gitlab/cwl-workflows/workflows/output/ubuntu.sif",
"checksum": "sha1$8a13313f5de5ace0d943ff7a3257fc83c0538829",
"size": 27742208,
"path": "/home/tom/gitlab/cwl-workflows/workflows/output/ubuntu.sif"
}
]
}
INFO Final process status is success
```
## Workflow Code
CommandLineTool `arv-get.cwl`:
```yaml
cwlVersion: v1.2
class: CommandLineTool
requirements:
DockerRequirement:
dockerPull: arv-cli
NetworkAccess:
networkAccess: true
InitialWorkDirRequirement:
listing:
- entryname: cwl.inputs.json
entry: '{"inputs": $(inputs)"}'
baseCommand:
- arv-get-cwl
inputs:
arvados_collection_locator: string
arvados_api_token: string
arvados_api_host: string
outputs:
collection_file: File
```
The `arv-get-cwl` script within the container extracts the input from `cwl.inputs.json` and passes it to the `arv-get` command, after which the `cwl.output.json` file is produced with the filename:
```bash
cat > ${outputfile} <
Contributor guide
Assessment
This issue has not been assessed yet.