bottlerocket-os / bottlerocket-os/bottlerocket

Custom CA in pki setting occasionally fails

Open
#4,556 8 comments 1 reaction 0 assignees View on GitHub
status/needs-triage type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**

Bottlerocket OS 1.39.0 (aws-k8s-1.32)

**What I expected to happen:**

We use a bootstrap container to download and configure an internal ca certificate along with proxy settings so we can pull all container images through a caching proxy.

```sh
#!/bin/bash
apt-get update
apt-get install -y curl
ca=$(curl -s http://our.internal.proxy.com/ca.crt | base64 -w 0)
echo "$ca"
/.bottlerocket/rootfs/bin/apiclient set --json "{\"pki\": {\"internal-ca\": {\"data\": \"$ca\", \"trusted\": true}}, \"network\": {\"https-proxy\": \"https://our.internal.proxy.com/\", \"no-proxy\": [\"172.20.0.0/16\", \"localhost\", \"127.0.0.1\", \".amazonaws.com\", \"169.254.169.254\", \"10.0.0.0/8\", \".internal\"]}}"
```

Most of the time, this works without issue. The ca cert is configured and images pull from our proxy successfully!

**What actually happened:**

Occasionally, the above process "works", but image pulls fail with a bad issuer cert error message. If we inspect the settings on the bottlerocket node, we do see that the ca configuration is in place, but it doesn't seem to be working. The bootstrap container itself is set as essential. so any failures in the above script should fail the node, but it doesn't seem to matter in this case since we can visually confirm the settings are in place on the node.

I am not sure what steps I can take to troubleshoot the issue, but I am happy to collect further information the next time we get a node in this state.

**How to reproduce the problem:**

Wait patiently for one of the nodes to trip on the error condition.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.