CWLProv roundtrip issues

> 117 tests passed, 16 failures, 0 unsupported features

Many of these do not give good error messages, so that should also be improved.
7 of these failures are also seen without a roundtrip, but they still need to be fixed (see https://github.com/common-workflow-language/cwltool/issues/916)

Failing tests
- [ ] 1 General test of command line generation
```
caused by: failed comparison for key 'args': expected: [
"bwa",
"mem",
"-t",
"2",
"-I",
"1,2,3,4",
"-m",
"3",
"chr20.fa",
"example_human_Illumina.pe_1.fastq",
"example_human_Illumina.pe_2.fastq"
]
got: [
"bwa",
"mem",
"-t",
"2",
"-I",
"1,2,3,4",
"-m",
"3",
"chr20.fa",
"chr20.fa",
"chr20.fa"
]
caused by: expected: "example_human_Illumina.pe_1.fastq"
got: "chr20.fa"
```

- [ ] 2 Test nested prefixes with arrays
```
caused by: failed comparison for key 'args': expected: [
"bwa",
"mem",
"chr20.fa",
"-XXX",
"-YYY",
"example_human_Illumina.pe_1.fastq",
"-YYY",
"example_human_Illumina.pe_2.fastq"
]
got: [
"bwa",
"mem",
"chr20.fa",
"-XXX",
"-YYY",
"chr20.fa",
"-YYY",
"chr20.fa"
]
caused by: expected: "example_human_Illumina.pe_1.fastq"
got: "chr20.fa"
```
- [ ] 55 Test InitialWorkDirRequirement linking input files and capturing secondaryFiles
Noted to fail without the roundtrip, see https://github.com/common-workflow-language/cwltool/issues/916
Still needs fixing
```
Must have class:File: {'basename': 'input.txt_idx8', 'location': 'file:///tmp/tmphkf70f2_/input.txt_idx8', 'class': 'Directory'}
```
- [x] ~~56 Test InitialWorkDirRequirement with expression in filename.~~
was error with re-using the `--outdir`, now fixed with an updated testing script
```
+ cwltool --provenance /tmp/tmp.fsSkdJkgCS --outdir=/tmp/tmp9vkms8v5 --quiet v1.0/rename.cwl v1.0/rename-job.json
+ cwltool --outdir=/tmp/tmp9vkms8v5 --quiet /tmp/tmp.fsSkdJkgCS/workflow/packed.cwl /tmp/tmp.fsSkdJkgCS/workflow/primary-job.json
Unhandled error, try again with --debug for more information:
[Errno 13] Permission denied: '/tmp/tmp9vkms8v5/fish.txt'
```
- [ ] 61 Test parameter evaluation, no support for JS expressions
Noted to fail without the roundtrip, see https://github.com/common-workflow-language/cwltool/issues/916
Still needs fixing
```
caused by: failed comparison for key 'bar': expected: {
"b az": 2,
"b\"az": null,
"b'az": true,
"baz": "zab1",
"buz": [
"a",
"b",
"c"
]
}
got: {
"@id": "urn:uuid:fccd103f-da84-4806-9d7a-691816e5df90",
"b az": 2,
"b\"az": null,
"b'az": true,
"baz": "zab1",
"buz": [
"a",
"b",
"c"
]
}
caused by: unexpected key '@id'
```
- [ ] 62 Test parameter evaluation, with support for JS expressions
Noted to fail without the roundtrip, see https://github.com/common-workflow-language/cwltool/issues/916
Still needs fixing
```
caused by: failed comparison for key 'bar': expected: {
"b az": 2,
"b\"az": null,
"b'az": true,
"baz": "zab1",
"buz": [
"a",
"b",
"c"
]
}
got: {
"@id": "urn:uuid:eb766103-e8fa-4d62-aa41-087c04b19797",
"b az": 2,
"b\"az": null,
"b'az": true,
"baz": "zab1",
"buz": [
"a",
"b",
"c"
]
}
caused by: unexpected key '@id'
```
- [ ] 65 Test format checking against ontology using subclassOf.
```
Expected value of 'input' to have format http://edamontology.org/format_2330 but
File has an incompatible format: {
"class": "File",
"location": "file:///tmp/tmp.kkMQ2dLVkY/data/ae/aeb3d11bdf536511649129f4077d5cda6a324118",
"format": "http://edamontology.org/format_1929",
"size": 12010,
"basename": "ref.fasta",
"nameroot": "ref",
"nameext": ".fasta",
"checksum": "sha1$aeb3d11bdf536511649129f4077d5cda6a324118",
"@id": "urn:uuid:6c8a24de-f9f6-490a-a279-d4c74ff0810b"
}
```
- [ ] 66 Test format checking against ontology using equivalentClass.
```
Expected value of 'input' to have format http://galaxyproject.org/formats/fasta but
File has an incompatible format: {
"class": "File",
"location": "file:///tmp/tmp.NF0674Y6bQ/data/ae/aeb3d11bdf536511649129f4077d5cda6a324118",
"format": "http://edamontology.org/format_1929",
"size": 12010,
"basename": "ref.fasta",
"nameroot": "ref",
"nameext": ".fasta",
"checksum": "sha1$aeb3d11bdf536511649129f4077d5cda6a324118",
"@id": "urn:uuid:b2428475-386b-42c4-992e-2000bc5904c7"
}
```
- [ ] 84 Test directory input with parameter reference
```
caused by: failed comparison for key 'outlist': expected: {
"checksum": "sha1$13cda8661796ae241da3a18668fb552161a72592",
"class": "File",
"location": "output.txt",
"size": 20
}
got: {
"basename": "output.txt",
"checksum": "sha1$4a48489b12a2fa852115d3ff8a94e2ee181084b2",
"class": "File",
"location": "file:///tmp/tmpnz7bpmjx/output.txt",
"path": "/tmp/tmpnz7bpmjx/output.txt",
"size": 10
}
caused by: field 'size' failed comparison: expected: 20
```
- [ ] 85 Test directory input in Docker
```
caused by: failed comparison for key 'outlist': expected: {
"checksum": "sha1$13cda8661796ae241da3a18668fb552161a72592",
"class": "File",
"location": "output.txt",
"size": 20
}
got: {
"basename": "output.txt",
"checksum": "sha1$e43e58309182b8ea4cda5810613f37d04f523410",
"class": "File",
"location": "file:///tmp/tmp613v8k7u/output.txt",
"path": "/tmp/tmp613v8k7u/output.txt",
"size": 6
}
caused by: field 'size' failed comparison: expected: 20
got: 6
```
- [ ] 87 Test directories in secondaryFiles
Noted to fail without the roundtrip, see https://github.com/common-workflow-language/cwltool/issues/916
Still needs fixing
```
Must have class:File: CommentedMap([('class', 'Directory'), ('basename', 'xtestdir'), ('location', 'file:///home/mcrusoe/common-workflow-language/v1.0/
v1.0/testdir')])
```
- [ ] 88 Test dynamic initial work dir
```
caused by: failed comparison for key 'outlist': expected: {
"checksum": "sha1$13cda8661796ae241da3a18668fb552161a72592",
"class": "File",
"location": "output.txt",
"size": 20
}
got: {
"basename": "output.txt",
"checksum": "sha1$0bf54b303eb2821a2116f334cbe183d38f47bf01",
"class": "File",
"location": "file:///tmp/tmpob29fdh7/output.txt",
"path": "/tmp/tmpob29fdh7/output.txt",
"size": 12
}
caused by: field 'size' failed comparison: expected: 20
got: 12
```
- [ ] 90 Test file literal as input
`[Errno 2] No such file or directory: '_:519b8115-d06f-44cd-86ec-0a74964c0779'`
Noted to fail without the roundtrip, see https://github.com/common-workflow-language/cwltool/issues/916
Still needs fixing
- [ ] 93 Test directory input with inputBinding
```
caused by: failed comparison for key 'outlist': expected: {
"checksum": "sha1$13cda8661796ae241da3a18668fb552161a72592",
"class": "File",
"location": "output.txt",
"size": 20
}
got: {
"basename": "output.txt",
"checksum": "sha1$4a48489b12a2fa852115d3ff8a94e2ee181084b2",
"class": "File",
"location": "file:///tmp/tmpogtpr42e/output.txt",
"path": "/tmp/tmpogtpr42e/output.txt",
"size": 10
}
caused by: field 'size' failed comparison: expected: 20
got: 10
```
- [x] ~~98 Test output of InitialWorkDir~~
was error with re-using the `--outdir`, now fixed with an updated testing script
`[Errno 13] Permission denied: '/tmp/tmpsxd92ges/ref.fasta'`
- [ ] 100 Test secondaryFiles on array of files.
```
ro_dir=/tmp/tmp.W32FbGrI2D
Input object failed validation:
../../../../tmp/tmp.W32FbGrI2D/workflow/primary-job.json:5:13: [Errno 2] No such file or directory: '/tmp/tmp.W32FbGrI2D/data/ae/ref.fasta.fai'
```
- [x] ~~101 Test directory literal output created by ExpressionTool~~
was error with re-using the `--outdir`, now fixed with an updated testing script
```
ro_dir=/tmp/tmp.rRYRCreLj6
+ cwltool --provenance /tmp/tmp.rRYRCreLj6 --outdir=/tmp/tmplnx5up31 --quiet v1.0/dir7.cwl v1.0/dir7.yml
+ cwltool --outdir=/tmp/tmplnx5up31 --quiet /tmp/tmp.rRYRCreLj6/workflow/packed.cwl /tmp/tmp.rRYRCreLj6/workflow/primary-job.json
Unhandled error, try again with --debug for more information:
[Errno 13] Permission denied: '/tmp/tmplnx5up31/a_directory/hello.txt'
```
- [x] 107 Test if a writable input directory is recursively copied and writable
```
caused by: {
"basename": "a",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"class": "File",
"location": "work_dir/a",
"size": 0
} not found
```
- [x] ~~117 Test empty writable dir with InitialWorkDirRequirement~~
was error with re-using the `--outdir`, now fixed with an updated testing script
` 'posix.DirEntry' object has no attribute 'startswith'`
- [x] ~~118 Test empty writable dir with InitialWorkDirRequirement inside Docker~~
was error with re-using the `--outdir`, now fixed with an updated testing script
` 'posix.DirEntry' object has no attribute 'startswith'`
- [ ] 120 Test file literal as input without Docker
Noted to fail without the roundtrip, see https://github.com/common-workflow-language/cwltool/issues/916
Still needs fixing
` [Errno 2] No such file or directory: '_:c2399658-cf22-42b8-bbd1-69b3fb8e22b8'`

Testing script, `cwlro_roundtrip.sh`
``` bash
#!/bin/bash
ro_dir=$(mktemp -d)

array=( "$@" )
unset "array[${#array[@]}-1]" # remove last element
unset "array[${#array[@]}-1]" # remove last element

declare -a ARGS
for var in "$@"; do # strip out outdir
if [[ "$var" == --outdir* ]]; then
continue
fi
ARGS[${#ARGS[@]}]="$var"
done

temp_outdir=$(mktemp -d)

cwltool --provenance ${ro_dir} --outdir=${temp_outdir} "${ARGS[@]}" > /dev/null
rm -Rf temp_outdir
cwltool ${array[@]} ${ro_dir}/workflow/packed.cwl ${ro_dir}/workflow/primary-job.json
```
Usage: `./run_test.sh RUNNER=$PWD/cwlro_roundtrip.sh`