cilium / cilium/cilium-cli

port-forward uses a SPDY-only dialer and fails behind proxies that do not support SPDY

Open
#3,258 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Makefile
Stars
597
Forks
227
Avg merge
2d 1h
Merged PRs (30d)
16

Description

Bug report

General Information

  • Cilium CLI version: v0.16.24 (the code below is unchanged on main and in v0.19.4)
  • Kubernetes: kubectl v1.31+, any apiserver version
  • Platform: environment-independent — reproduces whenever an HTTP proxy / gateway /
    load balancer that does not support SPDY sits in front of the kube-apiserver

Problem

cilium port-forward (e.g. cilium hubble port-forward) fails with
error upgrading connection: ... Upgrade request required when an HTTP proxy that
does not support SPDY is in front of the kube-apiserver.

The dialer in pkg/k8s/portforward/portforward.go (vendored from cilium/cilium)
is SPDY-only, with no WebSocket attempt and no fallback:

roundTripper, upgrader, err := spdy.RoundTripperFor(pf.config)
dialer := spdy.NewDialer(upgrader, &http.Client{Transport: roundTripper}, http.MethodPost, req.URL())

SPDY has been deprecated since 2015 (KEP-4006) and is no longer supported by many
proxies/gateways/load balancers. kubectl made WebSocket the default for port-forward
in v1.31, with SPDY as a fallback.

Why exec is unaffected

The exec path already uses the apimachinery WebSocket/SPDY fallback dialer
(createDialer, added in #37538). port-forward was never migrated.

#38988 later set exec to SPDY-primary with WebSocket fallback, due to hardcoded
idle-timeout issues in the upstream remotecommand WebSocket executor. That is
specific to remotecommand and does not apply to port-forward, which uses a separate
path (portforward.NewSPDYOverWebsocketDialer / the tunneling dialer).

Fix

Use the apimachinery fallback dialer in portforward.go with WebSocket as primary
and SPDY as fallback, matching kubectl's port-forward behaviour since v1.31:

spdyRoundTripper, upgrader, err := spdy.RoundTripperFor(pf.config)
if err != nil {
    return nil, err
}
spdyDialer := spdy.NewDialer(upgrader, &http.Client{Transport: spdyRoundTripper}, http.MethodPost, req.URL())

tunnelingDialer, err := portforward.NewSPDYOverWebsocketDialer(req.URL(), pf.config)
if err != nil {
    return nil, err
}

dialer := portforward.NewFallbackDialer(tunnelingDialer, spdyDialer, func(err error) bool {
    return httpstream.IsUpgradeFailure(err) || httpstream.IsHTTPSProxyError(err)
})

SPDY remains an automatic fallback for older API servers, so the change is
backward-compatible.

How to reproduce

  1. Place an HTTP proxy that does not support SPDY in front of the kube-apiserver.
  2. Run cilium hubble port-forward (or any cilium command that port-forwards).
  3. The SPDY upgrade is rejected and the command fails.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/k8s/portforward/portforward.go, where the SPDY-only dialer is created, and compare the exec path's createDialer with Kubernetes port-forward behavior. Verify the WebSocket-primary, SPDY-fallback behavior through an HTTP proxy that rejects SPDY, while preserving compatibility with older API servers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cli, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.