GoogleCloudPlatform / GoogleCloudPlatform/container-engine-accelerators

Downloading driver fails on a K8S 1.18 GKE Cluster

Open
#177 23 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
252
Forks
184
Avg merge
1h 55m
Merged PRs (30d)
2

Description

Using the [daemonset-nvidia-v450.yaml](https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/nvidia-driver-installer/cos/daemonset-nvidia-v450.yaml) fails due to a 403 error in a cluster with version `1.18.14-gke.1200`. [daemonset-preloaded.yaml](https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml) works fine in an 1.17 cluster but also fails when using an 1.18 cluster.

I've only captured the log of the v450 installer:

```
+ COS_KERNEL_INFO_FILENAME=kernel_info
+ COS_KERNEL_SRC_HEADER=kernel-headers.tgz
+ TOOLCHAIN_URL_FILENAME=toolchain_url
+ TOOLCHAIN_ENV_FILENAME=toolchain_env
+ TOOLCHAIN_PKG_DIR=/build/cos-tools
+ CHROMIUMOS_SDK_GCS=https://storage.googleapis.com/chromiumos-sdk
+ ROOT_OS_RELEASE=/root/etc/os-release
+ KERNEL_SRC_HEADER=/build/usr/src/linux
+ NVIDIA_DRIVER_VERSION=450.51.06
+ NVIDIA_DRIVER_MD5SUM=
+ NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia
+ NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia
+ ROOT_MOUNT_DIR=/root
+ CACHE_FILE=/usr/local/nvidia/.cache
+ LOCK_FILE=/root/tmp/cos_gpu_installer_lock
+ LOCK_FILE_FD=20
+ set +x
[INFO 2021-01-22 22:11:36 UTC] PRELOAD: false
[INFO 2021-01-22 22:11:36 UTC] Running on COS build id 13310.1041.38
[INFO 2021-01-22 22:11:36 UTC] Data dependencies (e.g. kernel source) will be fetched from https://storage.googleapis.com/cos-tools/13310.1041.38
[INFO 2021-01-22 22:11:36 UTC] Checking if this is the only cos-gpu-installer that is running.
[INFO 2021-01-22 22:11:36 UTC] Checking if third party kernel modules can be installed
/tmp/esp /
/
[INFO 2021-01-22 22:11:36 UTC] Checking cached version
[INFO 2021-01-22 22:11:36 UTC] Cache file /usr/local/nvidia/.cache not found.
[INFO 2021-01-22 22:11:36 UTC] Did not find cached version, building the drivers...
[INFO 2021-01-22 22:11:36 UTC] Downloading GPU installer ...
/usr/local/nvidia /
[INFO 2021-01-22 22:11:37 UTC] Downloading from https://storage.googleapis.com/nvidia-drivers-eu-public/nvidia-cos-project/85/tesla/450_00/450.51.06/NVIDIA-Linux-x86_64-450.51.06_85-13310-1041-38.cos
[INFO 2021-01-22 22:11:37 UTC] Downloading GPU installer from https://storage.googleapis.com/nvidia-drivers-eu-public/nvidia-cos-project/85/tesla/450_00/450.51.06/NVIDIA-Linux-x86_64-450.51.06_85-13310-1041-38.cos
curl: (22) The requested URL returned error: 403
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.