[Bug Report] Netkit device interface index drift causes silent eBPF redirect failure after hours of uptime
- Dominant language
- Go
- Stars
- 6.2k
- Forks
- 402
- Avg merge
- 1h 8m
- Merged PRs (30d)
- 1
Description
## Environment
- **dae version**: v2.0.0rc1+ (commit 028627d, based on PR#1016 9a6efd0)
- **Kernel**: 6.12.87 (ImmortalWrt 25.12, x86_64)
- **Deployment mode**: Transparent proxy / Netkit (kernel 6.7+)
## Summary
After running for several hours without any reload or configuration changes, dae suddenly stops proxying foreign traffic. Local (CN) sites work fine. This issue is **not** related to reload races (PR#1013/PR#1016) - it occurs spontaneously at runtime.
## Symptoms
1. ✅ dae process is healthy (PID unchanged, ~1.6GB RSS)
2. ✅ Node health checks all pass (s4=60ms, s2=170ms, s5=185ms via TCP connectivity check)
3. ✅ `dae0` interface exists and is UP
4. ❌ Foreign (proxy) traffic returns `Operation not permitted`
5. ✅ **Full restart (`/etc/init.d/dae restart`) fixes it immediately**
6. ✅ Reload test (20 iterations) passes fine after restart
## Root Cause Analysis
### How the Data Path Works
dae v2.0 creates a netkit device pair:
- `dae0` in the host network namespace
- `dae0peer` in a dedicated `daens` namespace
eBPF TC programs on `eth0` intercept traffic and use `bpf_redirect_peer()` to forward to `dae0`. The **interface index of dae0 is stored in a BPF map** at setup time.
### The Index Drift
Evidence from multiple failures:
| Occurrence | dae0 interface | Peer |
|------------|---------------|------|
| After first deploy | `dae0@if83` | if83 in daens |
| After first restart | `dae0@if85` | if85 in daens |
| After second restart | `dae0@if87` | if87 in daens |
When the kernel briefly deletes and recreates `dae0` (possibly due to memory pressure, udev/hotplug events, or sysctl triggers), the interface index changes. However, the **BPF map still holds the old index**. The eBPF program redirects to a stale/non-existent index → `EPERM`.
### Why This Is Silent
| Component | Status | Reason |
|-----------|--------|--------|
| dae process itself | ✅ OK | Health checks are internal connections, bypass eBPF redirect |
| dae0 interface | ✅ Exists (new) | But the BPF map has the old ifindex |
| Foreign traffic | ❌ EPERM | eBPF redirects to stale ifindex |
| dae logs | ❌ No errors | The error is in the eBPF program, not in userspace |
| Restart | ✅ Fixes | Re-creates dae0, BPF map gets the new ifindex |
## Logs
**Normal traffic (before failure):**
```
[2026-06-21 14:02:32] INFO 192.168.2.33:60126 <-> firebaseremoteconfigrealtime.googleapis.com:443 dialer=s4 ...
[2026-06-21 14:08:05] INFO 192.168.2.55:59896 <-> www.youtube.com:443 dialer=s4 ...
```
**Failure period (17:01-20:20):** Only health check logs (all nodes alive), **zero user traffic** through eBPF redirect.
**After restart:**
```
[2026-06-21 20:24:51] INFO 192.168.2.55:50523 <-> github.com:443 dialer=s4 ...
```
## Possible Root Causes
1. **Kernel memory reclaim** on tmpfs (488MB with dae using ~1.6GB) causes the netkit device to be reset
2. **udev / procd hotplug event** triggers interface reset
3. **Netkit device driver issue** in kernel 6.12.x
4. **sysctl / network configuration change** from other services
## Suggested Fixes
1. **Monitor and recover**: Detect `bpf_redirect_peer()` returning `EPERM` and trigger interface re-setup (refresh ifindex in BPF map)
2. **Netlink watch**: Use netlink notifications (`RTM_NEWLINK`/`RTM_DELLINK`) to detect dae0 recreation and update BPF map
3. **Periodic refresh**: Refresh BPF map with current dae0 ifindex every N minutes
4. **veth fallback option**: Add a config option to disable netkit and use veth (more mature device type)
## Workaround
```bash
# crontab health check every 10 minutes
*/10 * * * * wget -q -O /dev/null --timeout=10 https://www.google.com || /etc/init.d/dae restart
```
## Related Issues
- #1013 - Reload races during staged handoff (different root cause)
- #1016 - DNS config fingerprint for reload safety
This issue occurs **without any reload**, suggesting a runtime kernel/netkit interaction rather than a dae control plane race.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Netkit device setup and the point where dae0's interface index is stored in the eBPF map, then reproduce the failure without reload while monitoring link and map state. Done means confirming the index-drift path and adding a tested recovery or update mechanism that restores foreign-traffic proxying after dae0 is recreated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100