opensafely-core / opensafely-core/backend-server

Support ad-hoc docker execution in the VM's docker

Open
#85 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
1
Forks
5
Avg merge
1h 36m
Merged PRs (30d)
2

Description

The docker service in the Ubuntu VM is used by job-runner and can execute jobs submitted by job-runner.

However, some Bennett staff (researchers and tech team) needs to sometimes manually run docker. The tooling and documentation for this is based on using Docker for Windows.

So, we have kept Docker for Windows (with a much smaller resource allocation) around to continue to support this workflow.

This is not ideal:

a) Docker for Windows doesn't always come up after a reboot.
b) So that researchers can release outputs from these notebooks, we have to run hacky sync script from Windows host into the VM so that the TPP level 4 sync script can move them to level 4.
c) Would be nice to remove Docker for Windows and its resource usage.

The long term solution probably involves sqlrunner, but it's likely we'll still need an ad-hoc query mechanism.

We could point the docker client on the Windows host to the docker service running on the VM, perhaps over SSH auth. This would mean all users would need SSH set up.

However, while this would work for executing docker containers, we may need some trickery to mount workspaces in.

Another option is to just move all the tooling to require working over SSH in the VM. That might be reasonble, but add some additional friction to an already awkward process for researchers using this.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the VM's Docker service, its job-runner usage, and the linked notebook tooling and documentation. Compare the proposed host-to-VM Docker access and SSH-based workflow, including workspace mounts; done should provide ad-hoc container execution without Docker for Windows and its sync workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, ubuntu
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.