tailscale / tailscale/tailscale

FR: Add VXLAN-based packet mirroring to WireGuard data path

Open
#19,114 3 comments 0 reactions 0 assignees View on GitHub
fr needs-triage
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What are you trying to do?

Add optional, runtime-togglable VXLAN-based packet mirroring to Tailscale's WireGuard data path.
When enabled, a node replicates outbound encrypted WireGuard packet headers (or, optionally the full packet) to a standard VXLAN collector.
Mirroring is off by default, zero-cost when disabled, and can be included or excluded from builds via the `ts_omit_vxlanmirror` build tag.

The mirrored data enables observability of the tailscale packets:

| Signal | Source | Use case |
|---|---|---|
| Per-peer packet rate & throughput | WireGuard counter (nonce) + timestamps | Capacity planning, SLA monitoring |
| Packet loss & reordering | Counter gaps between consecutive packets | Debugging flaky links, ISP issues |
| Jitter | Inter-packet arrival time at collector | VoIP/video quality alerts |
| Handshake latency & failures | Initiation/Response timing, missing replies | NAT traversal debugging, key rotation monitoring |
| NAT rebinding | Endpoint address changes across handshakes | Diagnosing mobile roaming & carrier-grade NAT |
| DERP relay usage | Destination address reveals relay vs. direct | Identifying peers that can't establish direct paths |
| Session lifetime | Handshake rekey intervals | Security posture auditing |

### How should we solve this?

A possible implementation of "TailTap" is located [here](https://github.com/mlbright/tailscale/tree/tailtap) and has been lightly tested.

### Usage

```bash
# Start mirroring WireGuard headers to a collector
tailscale debug vxlan-mirror --dst 10.0.0.100:4789 --vni 1

# Check status
tailscale debug vxlan-mirror --status

# Escalate to full encrypted packet capture
tailscale debug vxlan-mirror --dst 10.0.0.100:4789 --full

# Stop
tailscale debug vxlan-mirror --stop
```

Collector side:

```bash
tcpdump -i eth0 -n udp port 4789 -w tailtap.pcap
```

It consists of a new `feature/vxlanmirror` package (~500 lines of implementation, ~700 lines of tests) that:

1. **Hooks into the magicsock send path** via a new `Engine.InstallMirrorHook` method.
The callback is stored in an `AtomicValue`; when nil (the default), the fast path is a single atomic load with no allocation.

2. **Encapsulates mirrored data in standard VXLAN** (RFC 7348) with synthetic Ethernet + IP + UDP inner headers.
Any tool that speaks VXLAN—Wireshark, Zeek, Suricata, ntopng, cPacket, Gigamon, AWS VPC Traffic Mirroring collectors—works out of the box with zero custom dissector plugins.

3. **Defaults to header-only mode**: only the 16-byte WireGuard transport header (message type + receiver index + counter) is sent per data packet.
Handshake packets (types 1–3) are mirrored in full since they carry no user data.
A `--full` flag sends the complete encrypted payload.

4. **Is controlled at runtime** via `tailscale debug vxlan-mirror` or `POST /localapi/v0/debug-vxlan-mirror`.
No daemon restart required to start, stop, or change the collector target.

5. **Is fully gated by `ts_omit_vxlanmirror`**.
Build with `-tags ts_omit_vxlanmirror` and the feature is dead-code-eliminated.
The `feature/buildfeatures` const-toggle pattern used by the rest of the codebase is followed exactly.

### Design highlights

- **Zero overhead when off**: the mirror hook is a nil `AtomicValue` check on the send path.
No branches, no allocations, no goroutines.
- **Concurrency-safe**: `Mirror` is protected by `sync.Mutex` for config and `atomic.Bool` for the full-packet toggle—safe to call from multiple goroutines on the hot path.
- **Correct VXLAN framing**: full IPv4/IPv6 inner headers with proper checksums (including mandatory UDP checksum for IPv6 per RFC 2460 §8.1), so collectors parse the frames without errors.
- **Follows existing patterns**: uses `feature.Register`, `localapi.Register`, build-tag gating via `feature/buildfeatures`, and `LocalBackend` integration via the `PacketMirror` interface—all consistent with how other optional features are structured.
- **Clean shutdown**: `LocalBackend.Shutdown()` tears down the mirror and uninstalls the engine hook.

### Files changed

| Path | Description |
|---|---|
| `feature/vxlanmirror/vxlanmirror.go` | Core Mirror implementation, VXLAN framing, local API handler |
| `feature/vxlanmirror/vxlanmirror_test.go` | Unit tests (header construction, checksums, capture length, status) |
| `net/packet/capture.go` | `MirrorCallback` type, `PacketMirror` interface |
| `wgengine/wgengine.go` | `InstallMirrorHook` added to `Engine` interface |
| `wgengine/magicsock/magicsock.go` | `mirrorHook` field, hook invocation in `Send()` |
| `wgengine/magicsock/endpoint.go` | Hook invocation in `endpoint.send()` |
| `wgengine/userspace.go` | Delegates `InstallMirrorHook` to magicsock |
| `wgengine/watchdog.go` | Pass-through `InstallMirrorHook` |
| `ipn/ipnlocal/vxlanmirror.go` | `LocalBackend` Start/Stop/Get/SetVXLANMirror methods |
| `cmd/tailscale/cli/debug-vxlanmirror.go` | CLI subcommand (build-tag gated) |
| `client/local/local.go` | Client library helpers for programmatic access |
| `feature/buildfeatures/feature_vxlanmirror_*.go` | Build-tag const toggle |
| `feature/condregister/maybe_vxlanmirror.go` | Conditional import |

### What is the impact of not solving this?

To be honest, nothing breaks if we never build or ship this feature.
Tailscale continues to work as it always has, and users can still debug connectivity issues with `tcpdump` on both endpoints.

Nevertheless, it can be useful to export WireGuard packets to external collectors without running packet captures on every node, especially in large deployments or large enterprises.

"Tailtap" enables users to:

- route packets to centralized collectors (Splunk, Elastic, Datadog, cPacket) alongside their existing network telemetry for fleet-wide dashboards.
- send audit trails that satisfy monitoring requirements in regulated environments (HIPAA, SOC 2, PCI-DSS).
- leverage the rich ecosystem of VXLAN-compatible tools (Wireshark, Zeek, Suricata, ntopng, Gigamon, cPacket, AWS VPC Traffic Mirroring collectors) for analysis without custom dissectors or plugins.

### Anything else?

- **Performance**: Header-only mode adds ~66 bytes of VXLAN framing per packet.
- **Security**: Only encrypted ciphertext is ever mirrored.
The destination is localhost/LAN-reachable only (no control-plane involvement).
Access is gated by `PermitWrite` on the local API.
- **Binary size**: Excluded entirely with `ts_omit_vxlanmirror`.
When included, adds ~500 lines of Go.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.