opensafely-core / opensafely-core/backend-server
Use systemd unix sockets to expose the db to running job containers
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 1
- Forks
- 5
- Avg merge
- 1h 36m
- Merged PRs (30d)
- 2
Description
Currently, when we run an ehrql job, we need to give it network access so it can access the db.
We don't want to give it unrestricted access, and the only way to do that is to create a custom docker network, and attach some iptables rules to that, then run the container with that network
This is all a bit complex and brittle, and the failure modes are likely insecure.
Instead, we could leverage systemd socket support and socat to expose the db over a unix socket, which we can then just mount into the ehrql container. There would no need to grant network access container, and we could remove all the docker network and iptables complexity.
Something like this should work:
db-proxy.socket:
Description=TPP db socket
[Socket]
ListenStream=/srv/high_privacy/run/db.socket
SocketUser=opensafely
SocketGroup=opensafely
SocketMode=0660
[Install]
WantedBy=sockets.target
db-proxy.service:
Description=TPP db proxy
Requires=db-proxy.socket
After=db-proxy.socket
[Service]
Type=simple
User=opensafely
Group=opensafely
ExecStart=/usr/bin/socat STDIO TCP:$DB_IP:$DB_PORT
StandardInput=socket
We are moving complexity around here, as we have a forked socat proxying each ehrql db connection. But socat is small, fast and battle tested reliable, so I think it may be a better solution than the current one. And I really like that we control network access via mounting explicit services as sockets, rather than allow but then block of the current approach.
We could also use this approach if we need other network services in containers, e.g. if we want to expose the otel-collector to allow db telemetry, we could just add a new socket proxy and mount that in too: https://github.com/opensafely-core/ehrql/issues/2481
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with services/jobrunner/sbin/jobrunner-network-config.sh, then inspect the referenced jobrunner/executors/local.py and jobrunner/lib/docker.py entry points. Compare the current Docker network and iptables setup with the proposed systemd socket and socat services. Done means database access can be provided through an explicitly mounted Unix socket without granting the job container network access, with the existing complexity removed safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, shell
- Domain
- devops, infrastructure, networking, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100