kubernetes-sigs / kubernetes-sigs/apiserver-network-proxy

DIAL_CLS / DIAL_RSP race leading to connection leak

Open
#404 5 comments 0 reactions 0 assignees View on GitHub
lifecycle/frozen
Dominant language
Go
Stars
449
Forks
225
Avg merge
3d 15h
Merged PRs (30d)
4

Description

There could be a race condition where a `DIAL_CLS` packet from the frontend is received at the same time as a `DIAL_RSP` from the backend that could lead to the backend connection being leaked:

This could happen if the following conditions happen in this order:

1. DIAL_RSP received from the backend
2. The pending dial is still present in https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/b5e5436b2fbeaa03657ff2381cef0d46f18ce267/pkg/server/server.go#L755
3. Frontend starts shutting down, sends a DIAL_CLS (prior to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/398 it wouldn't even send a close request)
4. Server sends the dial response the frontend - The FE gRPC stream is still open so the packet is received, but the frontend doesn't process it: https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/b5e5436b2fbeaa03657ff2381cef0d46f18ce267/pkg/server/server.go#L767
5. At this point, the server thinks the connection is established, but the frontend is not aware of that, and in the process of shutting down, leading to a leaked backend connection.

This seems fairly unlikely (at least once https://github.com/kubernetes-sigs/apiserver-network-proxy/issues/403 is fixed), but worth tracking.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/server/server.go around the pending-dial handling at the linked lines 755 and 767, and trace the DIAL_RSP and DIAL_CLS paths described in the issue. Reproduce or reason through the shutdown ordering, then verify that the race no longer leaves a backend connection leaked.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.