Azure / Azure/AKS

[BUG] Node provisioning fails behind an HTTP proxy: CSE exports only uppercase NO_PROXY, so APT ignores noProxy and proxies packages.microsoft.com

Open
#5,945 1 comment 0 reactions 1 assignee Claimed by @djsly View on GitHub
bug nodepools
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**

On a cluster with `httpProxyConfig`, node bootstrap fails whenever CSE has to install a package with
APT. `packages.microsoft.com` is in our `noProxy` list, but APT proxies it anyway.

The cause is a case mismatch. The CSE shell gets its proxy environment from `PROXY_VARS`, which
exports `http_proxy` (lowercase), `HTTPS_PROXY` (uppercase) and `NO_PROXY` (uppercase) — but never
`no_proxy`. **APT reads only the lowercase `no_proxy`.** Meanwhile `/etc/apt/apt.conf.d/95proxy`
sets a global APT proxy with no per-host exceptions, so the environment variable is APT's only
possible route to a bypass. The net effect is that `noProxy` has no effect on APT during CSE.

Where the proxy is an allowlisting mirror that cannot serve `packages.microsoft.com`, the
connection fails TLS, APT retries 10×, and CSE exits with `VMExtensionError_AptUpdateTimeout`. The
node never joins.

A cluster that hits this lands in `provisioningState: Failed` and **cannot create a node at all** —
no scale-out, no node-image upgrade, no auto-repair, no Kubernetes upgrade, no recovery from a lost
node. Rollback is equally blocked, because any `httpProxyConfig` write triggers the same node
rollout.

**To Reproduce**

Steps to reproduce the behavior:

1. Have an AKS cluster with `httpProxyConfig` set, where the configured proxy **cannot** serve
`packages.microsoft.com` (e.g. an allowlisting mirror, or a proxy with no matching certificate),
and where `noProxy` explicitly includes `packages.microsoft.com`.
2. Trigger any node creation — `az aks nodepool add`, `az aks nodepool scale`, or
`az aks nodepool upgrade --node-image-only`.
3. The VMSS instance fails CSE and is deleted; the operation returns
`VMExtensionError_AptUpdateTimeout`.

A faster reproduction that needs no node creation — run both on any existing node with
`httpProxyConfig`, same `apt.conf`, back to back:

```bash
env -u no_proxy -u http_proxy -u https_proxy NO_PROXY="packages.microsoft.com" \
/usr/lib/apt/apt-helper download-file \
https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease /tmp/upper
# -> Could not handshake: A TLS fatal alert has been received. FAILS (proxied)

env -u NO_PROXY -u http_proxy -u https_proxy no_proxy="packages.microsoft.com" \
/usr/lib/apt/apt-helper download-file \
https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease /tmp/lower
# -> Fetched 3632 B OK (bypassed)
```

Note that Azure deletes and recreates the VM after each CSE failure, so the failing node's log is
already gone by the time the operation reports failure. It has to be read live, mid-provision.

**Expected behavior**

Hosts listed in `httpProxyConfig.noProxy` should be bypassed by every bootstrap component that
honours proxy configuration, including APT during CSE. Node provisioning should succeed when the
required package hosts are excluded from the proxy.

**Screenshots**

No screenshots — verbatim output instead. The proxy address is replaced with `10.0.0.10:3128`
throughout; nothing else is altered.

CSE failure as reported by the VMSS extension:

```
CSE failed with 'VMExtensionError_AptUpdateTimeout'
Extracted CSE error: "Failed to fetch
https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease
Could not handshake: A TLS fatal alert has been received. [IP: 10.0.0.10 3128]"
```

Read off the node live, mid-provision:

```
NO_PROXY (upper): 1
no_proxy (lower): 0
+ timeout 180 apt-get -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/microsoft-prod.list ... update
+ '[' 5 -eq 10 ']' <- retry 5 of 10
+ sleep 5
```

Seen from the proxy, confirming the client is APT and the destination is a host listed in
`noProxy`:

```
"CONNECT - HTTP/1.1" 200 ... "Debian APT-HTTP/1.3 (2.4.14)" "packages.microsoft.com:443"
```

`/etc/apt/apt.conf.d/95proxy` in its entirety, as written by `configureEtcEnvironment()` — a global
proxy with zero per-host exceptions, and not one of the 25 `noProxy` entries anywhere:

```
Acquire::http::proxy "http://10.0.0.10:3128/";
Acquire::https::proxy "http://10.0.0.10:3128/";
```

**Environment (please complete the following information):**

- CLI Version: 2.82.0
- Kubernetes version: 1.34.8
- CLI Extension version: n/a (`aks-preview` not installed)
- Browser: n/a
- Node image: `AKSUbuntu-2204gen2containerd-202608.14.0` (also reproduces on `202605.14.0`)
- OS / APT: Ubuntu 22.04, APT 2.4.14
- Network: private cluster, `networkPlugin: none` (BYOCNI), `outboundType: userDefinedRouting`
- Proxy: cluster-wide `httpProxyConfig`, 25 `noProxy` entries including `packages.microsoft.com`

**Additional context**

### The inconsistency is inside AgentBaker itself

`getProxyVariables()` in
[`pkg/agent/variables.go`](https://github.com/Azure/AgentBaker/blob/main/pkg/agent/variables.go)
(duplicated in `aks-node-controller/parser/helper.go`) builds `PROXY_VARS` as:

```go
// from https://curl.se/docs/manual.html, curl uses http_proxy but uppercase for others?
proxyVars = fmt.Sprintf("export http_proxy=\"%s\";", ...) // lowercase
proxyVars = fmt.Sprintf("export HTTPS_PROXY=\"%s\"; %s", ...) // uppercase
proxyVars = fmt.Sprintf("export NO_PROXY=\"%s\"; %s", ...) // uppercase, no lowercase twin
```

and `cse_main.sh` confirms this is the CSE shell's only proxy environment:

```sh
# Setting vars in etc environment (configureEtcEnvironment) won't take effect in current shell session.
if [ -n "${PROXY_VARS}" ]; then eval $PROXY_VARS; fi
```

But `configureEtcEnvironment()` writes **both** cases to `/etc/environment`:

```sh
if [ -n "${NO_PROXY_URLS}" ]; then
echo "NO_PROXY=${NO_PROXY_URLS}" >> /etc/environment
echo "no_proxy=${NO_PROXY_URLS}" >> /etc/environment # lowercase is written here
fi
```

Verified on a live node: `/etc/environment` contains one `NO_PROXY=` and one `no_proxy=`. So the
lowercase form is known to be necessary — it is just missing from the one place that governs the
shell APT actually runs in. `PROXY_VARS` even lowercases `http_proxy`, so casing was clearly
considered for the proxy URL and not for the bypass list.

### Please note before closing: `/etc/environment` does not resolve this

`apt-get update` run manually on a booted node **works**, because a login shell and any unit with
`EnvironmentFile=/etc/environment` do get lowercase `no_proxy`. Only APT **during CSE** is affected.
That is very likely why this has gone unnoticed.

### Why this became reachable only recently

The uppercase-only export is not new — `pkg/agent/variables.go` has no proxy-related commit between
2025-03 and 2026-08. What changed is the consumer. Since roughly April 2026, kubelet, kubectl and
`azure-acr-credential-provider` are installed as APT packages from `packages.microsoft.com`
(#8287, #8292) rather than as curl-downloaded binaries. **curl honours uppercase `NO_PROXY`; APT
does not.** While everything used curl, the missing export was harmless.

It only bites on a package cache miss, since `installPkgWithAptGet` tries a cached `.deb` first.
`fallbackToKubeBinaryInstall` covers only kubelet and kubectl, so `azure-acr-credential-provider`
has no binary fallback — which matches the package we observed failing.

This is the same cache-miss shape as #5238, where the diagnosis was *"a recent refactoring which
caused the http proxy configuration to not be used when downloading an uncached k8s binary"*. That
one was the proxy not being used when it should be; this is the proxy being used when it should not.

### Suggested fix

Either would resolve it; the second is more robust as it does not depend on the shell environment.

1. **Export the lowercase form alongside the uppercase one** in `getProxyVariables()` — and in the
new `configureProxyEnvironment()` introduced by
[AgentBaker PR #9320](https://github.com/Azure/AgentBaker/pull/9320), which currently reproduces
the same uppercase-only pattern:
```sh
if [ -n "${NO_PROXY_URLS}" ]; then
export NO_PROXY="${NO_PROXY_URLS}"
export no_proxy="${NO_PROXY_URLS}" # APT reads only this one
fi
```

2. **Write the bypass list into APT's own configuration** in `configureEtcEnvironment()`, so it
holds regardless of environment:
```sh
for host in ${NO_PROXY_URLS//,/ }; do
echo "Acquire::http::Proxy::${host} \"DIRECT\";" >> /etc/apt/apt.conf.d/95proxy
echo "Acquire::https::Proxy::${host} \"DIRECT\";" >> /etc/apt/apt.conf.d/95proxy
done
```
CIDR and wildcard `noProxy` entries would need filtering here, as APT matches per host.

AgentBaker's e2e proxy coverage (`Test_Ubuntu2204_HTTPSProxy_PrivateDNS`) uses a real forwarding
proxy that will happily relay `packages.microsoft.com`, so the bypass never has to work for the test
to pass. A case where the proxy refuses a `noProxy` host would catch this.

### Impact

- Any cluster with `httpProxyConfig` whose proxy cannot reach `packages.microsoft.com` loses the
ability to create nodes as soon as a required package is not cached in the node image.
- Because the cache holds only recent versions, this arrives with no change by the customer — via
node-image auto-upgrade, a Kubernetes upgrade, or a reimage.

### Related

- Azure/AKS#5238 — same subsystem and same cache-miss trigger, opposite direction. Fixed via hotfix VHD.
- Azure/AgentBaker#9320 — open PR refactoring `PROXY_VARS` and `configureEtcEnvironment`; the natural
place for fix (1).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.