common-workflow-language / common-workflow-language/cwltool

[Proposal] customizable Job executor

Open
#748 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

Hello! I would like to propose a design for an extension to cwltool that allows the user to plug in customized job executors in addition to the existing ones (shell, docker).

The rationale for this is to integrate with services that provide code execution, like kubernetes, aws ec2, or some APIs internal to a company.

There are two possible ways to design this, in my opinion:

- Allowing to pass in a path to a python module that contains a subclass of `JobBase` that will be used instead of the existing ones. An instance of this job class will be returned by `CommandLineTool.makeJobRunner`.
- Implementing a new subclass of `JobBase`, called eg `ExternalJob`, that will be shipped inside `cwltool`, then allowing the user to pass in a module that will contain some methods that can be used by the `ExternalJob` class to start a job and check if a job is completed. ExternalJob will pass to the method all the required parameters to start a job (docker image, volumes, cwd, env vars, etc).

The advantage of the first approach is that it provides the greatest flexibility in terms of implementation. The advantage of the second approach is that it's higher level, so a lot of details are taken care of by the `ExternalJob` and the third party code would be easier to write.

I already have a POC for the second approach, and I would be very happy to collaborate to have such a mechanism being pushed upstream. I would understand if this is behind the scope of cwltool, but I personally think this is a very powerful addition.

cheers ✌️

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.