Properly mirror all packets (including kernel sockets) for mirrored network mode
- Dominant language
- C++
- Stars
- 33.7k
- Forks
- 1.8k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 116
Description
**Is your feature request related to a problem? Please describe.**
I have investigated this problem for a few weeks now and I realized all the problems I have, they all the following common symptoms:
1. No bidirectional traffic
2. Was able to intercept their respective packets in Wireshark from both side*, but it was not delivered to the WSL kernel eventually.
*: Given two machines A and B, if **WSL of A sends a packet to machine B**, the **packets are delivered to B and captured from Windows Wireshark side**, but **no captures from the tcpdump inside the WSL of B**
In addition, some examples are (Without loss of generality):
1. IPVS creates a virtual interface and connects to the backend servers from kernel itself. This surprisingly does not work.
2. Wireguard kernel module also does the same in a similar way, creates a UDP socket in kernel, connects to the peer by UDP from within kernel...
3. Any eBPF program also doesn't work
4. IP-over-UDP does not work
5. GRE does not work
6. VXLAN does not work
7. IPIP does not work
8. L2TPv3 was not tested but given the result in VXLAN, I assumed it won't work either
And thus, with all the aforementioned clues, and to my educated guesses, because those are all managed by the Linux kernel and not userspace, so if any userspace program works, then this means anything from kernel is clearly missing the mirroring somehow.
This theory is supported with this Wireguard workaround: https://github.com/microsoft/WSL/issues/10841#issuecomment-1831230148
I noticed there is something interesting with the following nftables rules (WLOG again):
```
table ip nat {
chain WSLPOSTROUTING {
type nat hook postrouting priority srcnat - 1; policy accept;
oif "eth0" udp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth0" tcp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth1" udp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth1" tcp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth2" udp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth2" tcp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth3" udp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth3" tcp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth4" udp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth4" tcp sport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
oif "eth4" udp dport 1-65535 meta mark != 0x00000001 counter packets 0 bytes 0 masquerade to :59600-59900
}
}
```
You can't reproduce those rules with normal nft commands, so I guess this is how mirrored mode was implemented at the moment (sadly the technical info on this was not transparent): by a combination of userspace ptrace (to rewrite the socket options and enable mirrored network support transparently), nftables, and Hyper-V networking hacks that zero-copies network packet from loopback device, then masquerade the network traffic to the main interface, all back and forth.
If my theory is correct, that means any kernel-initiated sockets, will end up forever stuck because it somehow bypassed the ptrace or somehow disregarded the nft rules, which means while the packets were delivered, it was missing some important information so that the Hyper-V network side did not realize this port was registered for mirroring from either incoming and outgoing side (and also sadly, not a true zero-copy packet mirror due to the masquerade which relied on stateful conntrack). And in the end the packet was eventually discarded on the Windows side due to missing receiver and timeout in the end.
To further prove this theory, I think IPSec would work if we separated the network connection to be handled by userspace Strongswan solely, and only offload TLS stuff to the kernel (I'm not sure if we can use KTLS that way). Another good proof would be testing whether KSMBD (kernel level Samba) works.
Related items:
#10841
#10840
https://github.com/microsoft/WSL/discussions/10730
**Describe the solution you'd like**
See if this theory is correct, and then check if anything necessary is missing for such scenarios, and try to implement them...
**Describe alternatives you've considered**
Shove it down your stomach and accept that mirrored network mode is for userspace only
**Additional context**
Unsurprisingly, this is a missing feature rather than a bug, because the people from MSFT clearly did not expect this kind of rare applications.
For us, we are running k0s, kube-proxy under IPVS mode, and VXLAN for Calico. We tried switching between IPIP, VXLAN, Wireguard and raw routing mode (that means `ip route ... via ` since all of our WSL machines are on the same L2 network) in Calico, none of that worked.
With a custom kernel and switching to iptables for kube-proxy mode, it worked somehow, but pod-to-pod communication is still unreachable and the conntrack table is full of 0-length packets under SYN_WAIT somehow.
### Rant time:
I'm highly aware that WSL is supposedly designated for a single-user based, semi-ephemeral developer environment, and not for running any critical server application like K8S after all.
However, as we have a huge sunk cost in Windows for our workstations, we cannot afford to switch to Linux, and add insult to the injury we need to use a lot of GPU resources for machine learning, but this is missing on the Windows side. We wanted to combine the best of both world by using kubeflow to manage the idling GPU resources efficiently.
And I'm very sure any MSFT consultants would suggest me to use an actual Hyper-V VM for this purpose, but we cannot afford to have DDA unfortunately for us being a small startup, so until MSFT have official support for GPU-P on Hyper-V VM (heck, even Windows GPU-P support is hidden behind secret Powershell commands and options), our only best bet is to use WSL2 and their official CUDA GPU support...with a bit of hack from me to gpu-operator as well.
We do have an ultimate workaround for all these: network bridge mode. This solves everything mentioned above, except this is recently "deprecated" in favor of the mirrored network mode...
Contributor guide
Research direction
Start with the related issues #10841 and #10840, then reproduce the reported failures for kernel-originated traffic such as WireGuard, VXLAN, GRE, or IPVS in mirrored network mode. Trace whether packets are delivered in both directions and determine what implementation work is required; done means the relevant kernel-created sockets and encapsulated traffic are mirrored successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, linux
- Domain
- infrastructure, networking, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100