Silent permanent disable of O_DIRECT streaming reads: no counters, verbose-only log
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
### Problem
In `cuda_model_stage_read()` (ds4_cuda.cu, currently around lines 2118-2127 on main), an `EINVAL`/`EFAULT`/`ENOTSUP`/`EOPNOTSUPP` from the O_DIRECT read path permanently disables direct I/O for the rest of the process:
```c
if (direct_errno == EINVAL || direct_errno == EFAULT || direct_errno == ENOTSUP || direct_errno == EOPNOTSUPP) {
if (getenv("DS4_CUDA_WEIGHT_CACHE_VERBOSE")) {
fprintf(stderr, "ds4: CUDA direct model read disabled: %s\n", strerror(direct_errno));
}
(void)close(g_model_direct_fd);
g_model_direct_fd = -1;
g_model_direct_align = 1;
}
```
The only evidence is a stderr line gated behind `DS4_CUDA_WEIGHT_CACHE_VERBOSE`. On any box or filesystem where this trips (alignment quirks, filesystems that reject O_DIRECT, FUSE/overlay setups), the streaming fast path silently degrades to buffered I/O forever, and nothing reports it. Users just see slower streaming, with no way to tell that direct I/O was ever engaged, let alone that it fell back.
### Proposal
Make the state and the fallback history observable, without changing the I/O behavior itself:
- counters for fallback events by errno class (einval / efault / enotsup / other-transient),
- the engaged/disabled/unavailable state plus the errno that caused a disable,
- widened-read stats (count + total wasted alignment bytes),
- an always-on log line on the permanent-disable event (a permanent behavior change should not be verbose-only),
- surfaced via `ds4_gpu.h` accessors and appended to the `--memory-report` output.
### Evidence the counters earn their keep
Running this instrumentation on a GB10 streaming deployment (DeepSeek V4 Flash, `--ssd-streaming`), the engaged path reports:
```
ds4: CUDA direct I/O: state=engaged disable_errno=0 fallbacks einval=0
efault=0 enotsup=0 other=0 widened_reads=69166 widen_wasted=270.179 MiB
avg_waste=4096.0 B/read
```
Every widened read pays exactly one extra 4 KB alignment page, ~0.03% of a typical ~13 MiB expert fetch -- the counters confirm the widening overhead is negligible, and the same surface makes a silent downgrade to buffered I/O immediately visible instead of invisible.
PR with the minimal patch attached.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Assessment
This issue has not been assessed yet.