globus / globus/globus-compute

Container Management

Open
#101 0 comments 0 reactions 3 assignees Claimed by @ZhuozhaoLi View on GitHub
enhancement help wanted
Dominant language
Python
Stars
162
Forks
53
Avg merge
15h 29m
Merged PRs (30d)
26

Description

Currently funcX defers the responsibility of container fetching to the user. However, we need to have better container management to handle the case where users might dynamically request containers not yet available on the resource.

Here's a sketch of v1 for Container management:
1. User registers a container specifying url and type:
`c_uuid = register(container_url, type='singularity|docker|shifter')`
2. Extra mechanism to manually register additional types against same container id
`a_bool = register(container_url, type='...', container_uuid=X)`
3. When user invokes a function tagged with a container uuid, the service issues the following
to the endpoint:
`func_uuid:container_uuid:`
4. The endpoint maintains a local table mapped to a json file in ~/funcx that contains, the
container_uuid: local_container_path mappings. The local mapping is either a container image
file on disk or some container url that the provider can resolve.
* `migrate_tasks_to_internal` checks the `container_uuid`, and if it is missing in the table, the endpoint agent has to put the task into an on_hold queue and triggers the container fetch process.
* If `container_uuid` is present, the uuid string is replaced by the local path.
4. Endpoint agents query the funcx_service for info on the container_uuid, and try to download the
appropriate container to the local disk and update the internal mapping. Once done, triggers a callback, that pops items in the on_hold tasks onto the pending queue with the container_uuid replaced by the local_path.

We want the container fetch to trigger a callback with the container_uuid so that the tasks matching that can be migrated to the pending queue. If instead a failure was encountered, we want an exception object to propagate back.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.