cockroachdb / cockroachdb/cockroach

server: file cache sizing should account for HighOpenFDCount alert threshold

Open
#171,403 1 comment 0 reactions 0 assignees View on GitHub
A-storage C-bug T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

The Pebble file cache is sized to consume nearly 100% of the OS file descriptor soft limit. When the cache is well-populated, the `HighOpenFDCount` alert (`sys_fd_open / sys_fd_softlimit > 0.8`) fires as a false positive. The alert is correct in that FD usage is high, but the high usage is expected and healthy behavior from the file cache.

## Background

CockroachDB sizes the file cache in `pkg/server/config_unix.go:setOpenFileLimitInner`:

```go
return (rLimit.Cur - recommendedNetworkFileDescriptors) / uint64(physicalStoreCount), nil
```

This reserves only 5,000 FDs for networking (`recommendedNetworkFileDescriptors`) and gives the rest to the file cache. With a 1M soft limit and 1 store, the file cache can hold ~995,000 entries, each consuming one FD. A well-populated cache plus networking FDs approaches 100% of the soft limit, well past the 80% alert threshold defined in `pkg/kv/kvserver/metric_rules.go:148-168`.

The file cache itself is functioning correctly (CLOCK-Pro eviction closes least-recently-used files when full). The problem is the sizing formula doesn't account for the alert's headroom requirement.

## Observed Impact

A customer with 443K SST files on a single store triggered `HighOpenFDCount` with no actual functional issue. The alert is not actionable and cannot be tuned by the customer (hardcoded threshold). See [support#3625](https://github.com/cockroachlabs/support/issues/3625).

## Proposed Fix

Scale the file cache size so that a full cache plus reserved FDs stays below the 80% alert threshold. For example:

```go
return (uint64(float64(rLimit.Cur)*0.75) - recommendedNetworkFileDescriptors) / uint64(physicalStoreCount), nil
```

This applies to all three return paths in `setOpenFileLimitInner` (lines 122, 129, 134).

An alternative is to adjust the alert threshold itself, but fixing the sizing is more robust since it also prevents FD exhaustion from a full cache in edge cases.

## References

- [support#3625](https://github.com/cockroachlabs/support/issues/3625)
- File cache creation: `pkg/server/config.go:776-781`
- FD limit calculation: `pkg/server/config_unix.go:26-135`
- Alert rule: `pkg/kv/kvserver/metric_rules.go:148-168`

Jira issue: CRDB-64508

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.