huggingface / huggingface/xet-core

Adaptive concurrency: RTT predictor emits degenerate predictions from near-singular fits

Open
#875 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
592
Forks
102
Avg merge
5d 8h
Merged PRs (30d)
9

Description

The adaptive concurrency RTT predictor (`xet_client/src/cas_client/adaptive_concurrency/`) can emit physically-impossible predictions when its weighted linear regression is near-singular — too few effective samples, or near-uniform transfer sizes at a single concurrency level. Filing to confirm whether this is a real controller bug: do these values feed concurrency decisions, or only reporting/logging?

### Mechanism
- **`predicted_bandwidth()` ≈ 9.5 TiB/s is the `1e-6` floor, not a measurement.** When `predicted_rtt(10 MiB, conc=1)` comes out negative it is clamped to 0 (`rtt_prediction.rs:94`), then `10 MiB / min_rtt.max(1e-6)` gives `10·1024·1024 / 1e-6 = 9.5367 TiB/s` (`rtt_prediction.rs:115-121`) — a data-independent constant.
- **`predicted_max_rtt` reaches tens of seconds**: the spurious regression slope extrapolated out to the 64 MiB reference query point.
- **The degenerate-fit signal is dropped.** `df = sw − 2 ≤ 0` returns `(Some(mean), None)` (`exp_weighted_olr.rs:86-87`), and the controller forwards the mean anyway via `.unwrap_or(0.)` (`controller.rs:143-145`).
- The singularity guard is **absolute** (`delta.abs() < 1e-12`, `exp_weighted_olr.rs:75`), so a tiny-but-nonzero `delta` (near-collinear x) passes and yields an exploding slope / large-negative intercept.

### Trigger
Low x-variance regime: early in a transfer, or uniform-size transfers at one concurrency level → the regression x-axis has almost no spread, so RTT jitter drives a near-vertical slope that is then extrapolated far past the observed size range.

### Suggested direction
Don't emit predictions outside the identifiable regime:
- Gate `predict()` on a **relative** condition number / minimum x-spread (not the absolute `1e-12`); return `None` for the mean when `df ≤ 0`.
- Have `predicted_bandwidth()` return `None` when the underlying RTT was clamped to 0, instead of dividing by the `1e-6` floor.
- Optionally cap extrapolation to near the observed size range.

Contributor guide

Open the contributing guide

Research direction

Start with exp_weighted_olr.rs:75-87 to trace the singularity and degrees-of-freedom handling, then follow predictions through rtt_prediction.rs:94 and 115-121 and controller.rs:143-145. Determine whether degenerate fits reach concurrency decisions or only reporting, and verify that low-variance inputs no longer produce clamped-zero RTTs, data-independent bandwidth, or implausibly extrapolated maximum RTTs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.