container-cache: emptyDir-backed cache never evicts because min_free reads the node filesystem
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- kubernetes, nginx
- Domain
- devops, infrastructure
Research direction
Start in the chart templates that render proxy_cache_path and the cache volume mounts, tracing the storageClassName: emptydir branch and its sizeProxyGB, sizeGB, and freeProxyPct values. Verify the rendered nginx configuration for emptyDir and persistentVolumeClaim modes; done means emptyDir zones include the proposed max_size accounting while persistent-volume deployments remain unchanged.
Written by the indexing model from the issue text.
Description
Problem
With persistentVolumeClaim.storageClassName: emptydir the chart mounts the cache directories as emptyDir volumes with sizeLimit set from sizeProxyGB / sizeGB, and drives nginx eviction with min_free only (no max_size on any proxy_cache_path).
An emptyDir has no filesystem of its own. nginx's min_free check runs statfs on the cache directory, which returns the free space of the node's ephemeral-storage volume (tens of TiB on a GPU node). A watermark of freeProxyPct percent of sizeProxyGB is therefore never reached, nginx never evicts, and the directory grows until kubelet enforces the emptyDir sizeLimit and evicts the whole pod. The cache restarts cold.
Observed while evaluating node-local NVMe as the cache backend: the node volume was 23.95 TiB, sizeProxyGB 1500, freeProxyPct 15, giving min_free=225g, about 1 percent of the volume.
Expected
In emptyDir mode nginx should evict by its own accounting before the emptyDir limit is hit, keeping freeProxyPct as the single knob meaning "start evicting at (100 - pct) percent usage".
Proposed fix
When storageClassName is emptydir, add max_size=(100 - freeProxyPct)% of the zone's configured size to each proxy_cache_path, keeping min_free as is. PersistentVolume deployments are unchanged.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 417
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/nvcf
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
good-first-issue
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Similar issues
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
githubnext/gh-aw-cao#13143 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
blinklabs-io/bursa#904 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
yanet-platform/ipfw-go#129 ·
-
bug confmap/provider/googlesecretmanagerprovider needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
open-telemetry/opentelemetry-collector-contrib#51273 · 2 comments ·
-
bug: AI Gateway client filter lists "Unknown" twice when NULL and literal Unknown clients coexist Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100