[Feature]: Support mounting a node-local (on-disk) package repository into the driver container
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
Context:
Nvidia driver daemonset deployed by GPU Operator (for non-pre-compiled driver mode) has a step to pull some Linux OS distro images at run-time (during driver pod bringup) from a Linux distro packages repository (e.g. an apt packages repository for Unbutu), location of this repository is over the network (e.g. by default, archive.ubuntu.com over the Internet, or in case of air-gap, a local mirror setup by the user, till over the network).
(The packages being fetched over the network and installed inside the driver pod, e.g. linux-kernel headers etc., are required to compile the driver from sources - in order to match the exact running kernel on the kubernetes cluster node/"host").
Problem:
Runtime issues:
Fetching packages over the network often creates a failure point any time the k8s cluster node is being "prepared" (before GPU using workload can run). Some failure scenarios (not exhaustive):
- Due to "Internet weather", archive.ubuntu.com is inaccessible
- Canonical's policy removes some older package versions (e.g. kernel headers package for an "older" version) from main repo (the default one) into snapshot
etc.
Operational issues:
This need for an Linux distro packages repo also complicates setting up air-gap environment - an additional mirror that the end-user needs to setup (in addition to mirroring the Nvidia container images).
Proposed Solution:
If the k8s cluster node/"host" provides the additional Linux distro OS packages on the node's file system itself (e.g. also bundle the matching kernel header package & other packages, for the kernel version being run by the node), then instruct GPU operator to fetch the needed packages from the node itself - a "node-local" I/O instead of network I/O.
This eliminates a class of problems describes above.
Implementation problem:
While GPU Operator today allows overriding the location of the Linux distro packages repo, e.g. say the GPU Operator chart's driver.repConfig.configMapName points to a configmap that has .data.value set as:
Types: deb
URIs: file:///opt/local-packages
Suites: ./
Signed-By: /opt/local-packages/keyring.gpg
currently GPU operator implicitly assumes that repo is accessed over the network, the driver daemonset pod/container ( that runs the driver install script which in turn triggers the fetching of packages) does not have access to the node/host's file system (the init container has it, but not the main container)
The template deployed by GPU Operator would need to allow "host path volume mount" in a specific manner (to avoid increasing attack surface), e.g. instead of arbitrary host mount config, have a specific config that mounts a read-only volume inside the pod/container.
Also this feature needs to be opt-in, if the provider of node image, of the k8s cluster that is going to run GPU Operator, is supplying this "node-local packages repo", then the user should be able to configure access to it through GPU Operator config API.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the GPU Operator chart's driver daemonset template, the driver.repConfig.configMapName configuration, and the driver install script entry point. Define the opt-in configuration and read-only host path mount needed for the node-local repository, then verify that packages can be fetched from the mounted path while existing network-based behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, linux
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100