cockroachdb / cockroachdb/cockroach
server: file cache sizing should account for HighOpenFDCount alert threshold
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Summary
The Pebble file cache is sized to consume nearly 100% of the OS file descriptor soft limit. When the cache is well-populated, the `HighOpenFDCount` alert (`sys_fd_open / sys_fd_softlimit > 0.8`) fires as a false positive. The alert is correct in that FD usage is high, but the high usage is expected and healthy behavior from the file cache.
## Background
CockroachDB sizes the file cache in `pkg/server/config_unix.go:setOpenFileLimitInner`:
```go
return (rLimit.Cur - recommendedNetworkFileDescriptors) / uint64(physicalStoreCount), nil
```
This reserves only 5,000 FDs for networking (`recommendedNetworkFileDescriptors`) and gives the rest to the file cache. With a 1M soft limit and 1 store, the file cache can hold ~995,000 entries, each consuming one FD. A well-populated cache plus networking FDs approaches 100% of the soft limit, well past the 80% alert threshold defined in `pkg/kv/kvserver/metric_rules.go:148-168`.
The file cache itself is functioning correctly (CLOCK-Pro eviction closes least-recently-used files when full). The problem is the sizing formula doesn't account for the alert's headroom requirement.
## Observed Impact
A customer with 443K SST files on a single store triggered `HighOpenFDCount` with no actual functional issue. The alert is not actionable and cannot be tuned by the customer (hardcoded threshold). See [support#3625](https://github.com/cockroachlabs/support/issues/3625).
## Proposed Fix
Scale the file cache size so that a full cache plus reserved FDs stays below the 80% alert threshold. For example:
```go
return (uint64(float64(rLimit.Cur)*0.75) - recommendedNetworkFileDescriptors) / uint64(physicalStoreCount), nil
```
This applies to all three return paths in `setOpenFileLimitInner` (lines 122, 129, 134).
An alternative is to adjust the alert threshold itself, but fixing the sizing is more robust since it also prevents FD exhaustion from a full cache in edge cases.
## References
- [support#3625](https://github.com/cockroachlabs/support/issues/3625)
- File cache creation: `pkg/server/config.go:776-781`
- FD limit calculation: `pkg/server/config_unix.go:26-135`
- Alert rule: `pkg/kv/kvserver/metric_rules.go:148-168`
Jira issue: CRDB-64508
Contributor guide
Assessment
This issue has not been assessed yet.