common-workflow-language / common-workflow-language/cwltool

Avoid downloading expression tool inputs and outputs to disk?

Open
#1,666 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I want to be able to run a tool like this, and let it output a `File` without having to have the data on the local disk:

```
#!/usr/bin/env cwl-runner
cwlVersion: v1.2
class: ExpressionTool
requirements:
InlineJavascriptRequirement: {}
inputs:
f1:
type: File
outputs:
f2:
type: File
expression: |
${ return {f2: f1}; }
```

This tool never accesses the file data, and I'm not sure that an `ExpressionTool` is even allowed to go around opening files from JavaScript, so there's no reason the file should need to be downloaded beforehand. I think in vanilla `cwltool` this would apply when the `location`(s) are HTTP(S) URLs, but in Toil we pass around `File` and `Directory` objects with a variety of URL schemes in `location`. Also, an ExpressionTool might or might not be allowed to just build a `{class: 'File'}` itself. So it makes sense for `File` objects to come out of a tool that don't correspond to things that are on disk.

But, at the end of a tool, the executor wants to run `relocateOutputs`, and that in turn seems to demand that the files be on disk unless the scheme is the magic `_:` one for files to be created:

https://github.com/common-workflow-language/cwltool/blob/898fddf9167620645177eb94c8ea68ede0667f96/cwltool/process.py#L397-L398

It looks like it does this for checksumming, but checksumming might not be turned on, and the checksum might already be set.

Is there a way to get a hook in here so that Toil can let `File` objects through expression tools without downloading them?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.