bottlerocket-os / bottlerocket-os/bottlerocket
Pods with `hostNetwork: true` timeout when querying kube api server
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 12
Description
We have deployed cloudwatch agent in the EKS cluster from its helm chart. The agent is deployed as daemonset, with:
`hostNetwork: true`
`dnsPolicy: ClusterFirstWithHostNet`
The cloudwatch agent pod logs show that it is timing out when trying to connect to the default kubernetes service for the kube api server.
`Error: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://:443/api/v1/namespaces/amazon-cloudwatch/services?labelSelector=k8s-app%3Ddcgm-exporter-service&limit=500&resourceVersion=0": dial tcp :443: i/o timeout"`
We have ruled out networking issues within the cluster as other pods can query the service and pods (including the cloudwatch agent) can query each others services.
We deploy the bottlerocket image with a bootstrap container which executes a script that adds an IPv6 address to the node network interface.
Following is the content of the script:
```
#!/bin/bash
set -eo pipefail
echo "Running bottlerocket bootstrap script"
set -x
USER_DATA_FILE=/.bottlerocket/bootstrap-containers/current/user-data
if [ -f "${USER_DATA_FILE}" ]; then
. "${USER_DATA_FILE}"
fi
function config_vip() {
if [ -z "${VIP}" ]; then
>&2 echo "IPv6 VIP has not been configured"
exit 1
fi
ip addr add dev eth0 local "${VIP}"
}
config_vip
exit 0
```
We believe the issue arises due to the addition of the above IP to the network interface.
We came to the conclusion after performing following tests:
- Connected to the bottle rocket node from session manager and ran:
`ip route get `
The result:
` from :: via dev eth0 proto ra src metric 1024 pref medium`
This suggests the assigned IP from the bootstrap container is used as source.
- Deleted the IPv6 assigned from the bootstrap container: `ip addr del dev eth0 local `
Ran `ip route get ` again.
This time the result was: ` from :: via dev eth0 proto ra src metric 1024 pref medium`
- After deleting the IP as above, the cloudwatch agent pod was able to connect to the kube api server
We have observed this behaviour accross different EKS clusters across different AWS accounts.
This issue did not arise when using Amazon Linux image.
**Image I'm using:**
bottlerocket-aws-k8s-1.32-x86_64-v1.32.0-cacc4ce9
**What I expected to happen:**
The cloudwatch agent pod should be able to query the kube api server without timeouts.
**What actually happened:**
The cloudwatch agent pod times out when querying the kube api server.
**How to reproduce the problem:**
Assign an IP to the bottlerocket node network interface and query the kube api server service from a pod that has `hostNetwork: true`.
Contributor guide
Research direction
Start with the bootstrap-container script and reproduce the hostNetwork pod request on the listed Bottlerocket image. Compare `ip route get` output and API-server connectivity before and after adding the IPv6 address, then inspect the relevant Bottlerocket networking behavior. Done means identifying a reproducible cause and a verified fix or documented limitation for the timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, bash, kubernetes, linux
- Domain
- cloud, networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100