microsoft / microsoft/retina

implement adaptive sampling instead of probablistic sampling

Open
#1,967 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3.2k
Forks
304
Avg merge
1d 19h
Merged PRs (30d)
78

Description

On high-throughput clusters, the current static `dataSamplingRate` configuration requires manual tuning and cannot respond to changing traffic conditions. Users must guess an appropriate sampling rate, too aggressive causes unnecessary data loss during quiet periods, too permissive causes buffer overflow during traffic spikes.

This current static approach cannot adapt to load changes, and when `retina_lost_events_total` starts climbing, there is no automatic mechanism to reduce event volume.

**Describe the solution you'd like**
related to the feature ask https://github.com/microsoft/retina/issues/1966

Implement adaptive sampling using BPF ring buffer back-pressure (requires kernel 5.8+). With ring buffers, `bpf_ringbuf_reserve()` returns NULL when the buffer is full, providing natural back-pressure without explicit sampling logic.

This approach provides:
- **Zero overhead** when buffer has capacity, no random number generation or map lookups per packet
- **Automatic adaptation** drops events only when buffer is actually full
- **Configurable capacity** users tune buffer size rather than sampling rate
- **Predictable behavior** buffer size directly controls memory usage and burst capacity

This should be implemented alongside the BPF ring buffer feature request, as it depends on `BPF_MAP_TYPE_RINGBUF` (kernel 5.8+).

**Describe alternatives you've considered**

1. **BPF map-based rate control** - Userspace monitors load and writes sampling rate to a BPF map that the BPF program reads per-packet. Adds map lookup overhead and has feedback delay between userspace detection and BPF adjustment.

2. **Token bucket in BPF** - Implement rate limiting entirely in BPF using per-CPU maps. Complex to implement correctly with per-CPU state management and token refill logic.

**Additional context**

This feature is tied to the BPF ring buffer implementation. Ring buffers provide natural back-pressure that eliminates the need for explicit adaptive sampling logic. The buffer itself becomes the adaptation mechanism. Users configure buffer size based on their memory budget and acceptable burst capacity, and the system automatically drops events only when that capacity is exceeded.

Reference: https://nakryiko.com/posts/bpf-ringbuf/ "BPF ring buffer provides a special BPF_RB_NO_WAKEUP flag that can be used to avoid waking up user-space when buffer space is available, as well as BPF_RB_FORCE_WAKEUP to force wake-up."

Related to https://github.com/microsoft/retina/issues/655 as that kickstarted our internal investigation

Contributor guide

Open the contributing guide

Research direction

Start by reading related issues 1966 and 655, then inspect the current dataSamplingRate handling and retina_lost_events_total behavior. Verify how the BPF ring buffer dependency and kernel 5.8+ requirement fit the existing implementation. Done should mean event volume adapts to ring-buffer capacity without manual sampling-rate tuning, with configurable buffer capacity and documented behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
networking, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.