common-workflow-language / common-workflow-language/cwltool

Input `format` expression not considering `$namespaces`

Open
#2,033 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior

When using a file `format` with a JS expression (workaround for https://github.com/common-workflow-language/cwl-v1.3/issues/52), the `format` check should consider any relevant resolution of `$namespaces` beforehand.

## Actual Behavior

The `format` check fails if the input file format and the evaluated format do not match *exactly*. Since `$namespaces` can be used to write equivalent formats (i.e.: `https://www.iana.org/assignments/media-types/application/geo+json` ↔ `iana:application/geo+json`), they should be considered interchangeably. However, the `format` expression fails unless the evaluated format is explicitly written in its long form (ie: the full URI).

Given that inputs submitted (in `job.yml`) either with the long-form URI or the namespace'd `format` are both converted to the long-form URI when reaching the below check, this forces the JS expression to use the long-form URI to be considered valid.
https://github.com/common-workflow-language/cwltool/blob/6d8c2a41e2c524e8d746020cc91711ecc3418a23/cwltool/builder.py#L555-L559

However, for a user writing the CWL document that defined a `$namespace` section, it is very counter-intuitive to use the long-form URI only in the `format` expression, when everywhere else accepts `iana:application/geo+json`.

## Workflow Code

`job.yml`
```yaml
features:
class: File
path: /tmp/feature-0.geojson
format: "https://www.iana.org/assignments/media-types/application/geo+json"
```
OR
```yaml
features:
class: File
path: /tmp/feature-0.geojson
format: "iana:application/geo+json"
```

`echo_features.cwl`
```cwl
cwlVersion: "v1.2"
class: CommandLineTool
$namespaces:
iana: "https://www.iana.org/assignments/media-types/"
baseCommand: echo
requirements:
InlineJavascriptRequirement: {}
DockerRequirement:
dockerPull: "debian:stretch-slim"
inputs:
features:
type:
- "File"
- type: array
items: File
format: |
${
if (Array.isArray(inputs.features)) {
return "iana:application/geo+json"; # (!) here is the problematic format, unless the full URI is given
}
return "http://www.opengis.net/def/glossary/term/FeatureCollection";
}
inputBinding:
valueFrom: |
${
if (Array.isArray(inputs.features)) {
return {
"type": "FeatureCollection",
"features": inputs.features.every(item => item.contents)
};
}
return inputs.features.contents;
}
outputs:
features:
type: File
format: "http://www.opengis.net/def/glossary/term/FeatureCollection"
outputBinding:
glob: "features.json"
stdout: "features.json"

```

## Full Traceback
```pytb
❯ cwllog --debug echo_features.cwl job.yml
Running: [cwltool --disable-color --debug echo_features.cwl job.yml 2>&1 | tee echo_features.log]
Log Path: [/tmp/echo_features.log]
INFO /home/francis/dev/conda/envs/weaver/bin/cwltool 3.1.20230906142556
INFO Resolved 'echo_features.cwl' to 'file:///tmp/echo_features.cwl'
URI prefix '${
if (Array.isArray(inputs.features)) {
return "iana' of '${
if (Array.isArray(inputs.features)) {
return "iana:application/geo+json";
}
return "http://www.opengis.net/def/glossary/term/FeatureCollection";
}
' not recognized, are you missing a $namespaces section?
URI prefix '${
if (Array.isArray(inputs.features)) {
return "iana' of '${
if (Array.isArray(inputs.features)) {
return "iana:application/geo+json";
}
return "http://www.opengis.net/def/glossary/term/FeatureCollection";
}
' not recognized, are you missing a $namespaces section?
echo_features.cwl:9:3: object id 'echo_features.cwl#features' previously defined
WARNING echo_features.cwl:22:7: JSHINT: "features": inputs.features.every(item => item.contents)
echo_features.cwl:22:7: JSHINT: ^
echo_features.cwl:22:7: JSHINT: W119: 'arrow function syntax (=>)' is only available in ES6. CWL only supports ES5.1
ERROR Workflow error:
Expected value of 'features' to have format '${\n if (Array.isArray(inputs.features)) {\n return "iana:application/geo+json";\n }\n return "http://www.opengis.net/def/glossary/term/FeatureCollection";\n}\n' but
File has an incompatible format: {
"class": "File",
"format": "https://www.iana.org/assignments/media-types/application/geo+json",
"location": "file:///tmp/feature-0.geojson",
"size": 162,
"basename": "feature-0.geojson",
"nameroot": "feature-0",
"nameext": ".geojson"
}
Traceback (most recent call last):
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/builder.py", line 554, in bind_input
check_format(
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwl_utils/file_formats.py", line 70, in check_format
raise ValidationException(
schema_salad.exceptions.ValidationException: File has an incompatible format: {
"class": "File",
"format": "https://www.iana.org/assignments/media-types/application/geo+json",
"location": "file:///tmp/feature-0.geojson",
"size": 162,
"basename": "feature-0.geojson",
"nameroot": "feature-0",
"nameext": ".geojson"
}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/main.py", line 1298, in main
(out, status) = real_executor(
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/executors.py", line 62, in __call__
return self.execute(process, job_order_object, runtime_context, logger)
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/executors.py", line 145, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/executors.py", line 218, in run_jobs
for job in jobiter:
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 963, in job
builder = self._init_job(job_order, runtimeContext)
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/process.py", line 888, in _init_job
builder.bind_input(
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/builder.py", line 330, in bind_input
self.bind_input(
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/builder.py", line 262, in bind_input
self.bind_input(
File "/home/francis/dev/conda/envs/weaver-py310/lib/python3.10/site-packages/cwltool/builder.py", line 560, in bind_input
raise WorkflowException(
cwltool.errors.WorkflowException: Expected value of 'features' to have format '${\n if (Array.isArray(inputs.features)) {\n return "iana:application/geo+json";\n }\n return "http://www.opengis.net/def/glossary/term/FeatureCollection";\n}\n' but
File has an incompatible format: {
"class": "File",
"format": "https://www.iana.org/assignments/media-types/application/geo+json",
"location": "file:///tmp/feature-0.geojson",
"size": 162,
"basename": "feature-0.geojson",
"nameroot": "feature-0",
"nameext": ".geojson"
}
```

## Your Environment
* cwltool version: 3.1.20230906142556

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.