common-workflow-language / common-workflow-language/common-workflow-language
Generalised ContainerRequirement
- Dominant language
- Common Workflow Language
- Stars
- 1.5k
- Forks
- 193
- PR merge metrics
- No merged PRs in 30d
Description
Docker is great, but other container packaging and execution tools exist and it would be unfortunate for workflow authors to require a specific dependency on docker when docker-specific features are not actually required.
I suggest that it would be advantageous for CWL to support a more generic `ContainerRequirement` which extends `ProcessRequirement` and that specifies the minimum set of parameters necessary to support containerisation without any docker-specific features. This could simply consist of a content address that describes the contents of the container rootfs along with a pointer to where/how to obtain one (i.e. a URL), along with specifying the semantics of how to map a CommandLineTool into the generic container (the ability to run a command line within a container seems like it ought to be fundamental to any container executor, but perhaps we need to be explicit about what the default environment should be). Both the content address and the URL could be optional (`required=False`).
The existing `DockerRequirement` would be retained, but modified to extend `ContainerRequirement` rather than extending `ProcessRequirement` directly. The purpose of `DockerRequirement` would be to specify docker-specific functionality, including how to obtain a container using docker pull/load/file/imageid.
It would be valid to specify _both_ `ContainerRequirement` _and_ `DockerRequirement` -- the semantics of this would depend on whether they are specified in `requirements` or `hints`. Here is one way it might work:
`ContainerRequirement` in `requirements`, `DockerRequirement` in `requirements`: specifically require the docker container executor, but after obtaining the docker image as specified in the `DockerRequirement`, calculate the rootfs content address and require that it matches the one specified in the `ContainerRequirement` (if specified).
`ContainerRequirement` in `requirements`, `DockerRequirement` in `hints`: require containerisation and that the rootfs matches the specified content-address, but the implementation can choose to obtain the image from the URL specified in `ContainerRequirement` _or_ the docker-specific mechanism specified in `DockerRequirement` (or in future from another container system's mechanism if it is hinted at as well).
`ContainerRequirement` in `hints`, `DockerRequirement` in `requirements`: specifically require docker and obtain the container following docker-specific semantics. Check the content address of the container if specified in `ContainerRequirement` but continue running (perhaps with a warning) if it doesn't match what is obtained via docker.
`ContainerRequirement` in `hints`, `DockerRequirement` in `hints`: the implementation is free to choose to use docker or another containerisation system, or to not use containerisation at all. Warnings may be issued if the `ContainerRequirement` rootfs content address does not match what is provided from docker.
In future, other specific container systems could be added to support specific semantics or container image discovery mechanisms, but the common requirement for `ContainerRequirement` can always be specified to enforce checking of the rootfs content-address.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the CWL definitions of ProcessRequirement and DockerRequirement, then map the proposed ContainerRequirement inheritance and the requirements-versus-hints combinations described here. Done means the generic container semantics and Docker-specific behavior are agreed and specified consistently; this issue leaves those design decisions unresolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100