kubernetes-sigs / kubernetes-sigs/apiserver-network-proxy
DIAL_CLS / DIAL_RSP race leading to connection leak
- Dominant language
- Go
- Stars
- 449
- Forks
- 225
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 4
Description
There could be a race condition where a `DIAL_CLS` packet from the frontend is received at the same time as a `DIAL_RSP` from the backend that could lead to the backend connection being leaked:
This could happen if the following conditions happen in this order:
1. DIAL_RSP received from the backend
2. The pending dial is still present in https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/b5e5436b2fbeaa03657ff2381cef0d46f18ce267/pkg/server/server.go#L755
3. Frontend starts shutting down, sends a DIAL_CLS (prior to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/398 it wouldn't even send a close request)
4. Server sends the dial response the frontend - The FE gRPC stream is still open so the packet is received, but the frontend doesn't process it: https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/b5e5436b2fbeaa03657ff2381cef0d46f18ce267/pkg/server/server.go#L767
5. At this point, the server thinks the connection is established, but the frontend is not aware of that, and in the process of shutting down, leading to a leaked backend connection.
This seems fairly unlikely (at least once https://github.com/kubernetes-sigs/apiserver-network-proxy/issues/403 is fixed), but worth tracking.
Contributor guide
Research direction
Start in pkg/server/server.go around the pending-dial handling at the linked lines 755 and 767, and trace the DIAL_RSP and DIAL_CLS paths described in the issue. Reproduce or reason through the shutdown ordering, then verify that the race no longer leaves a backend connection leaked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100