GoogleCloudPlatform / GoogleCloudPlatform/ops-agent
Ops Agent 2.50.0+: Excessive disk reads due to file_cache_advise default
- Dominant language
- Go
- Stars
- 204
- Forks
- 87
- Avg merge
- 21h 54m
- Merged PRs (30d)
- 23
Description
## Problem
Starting with Ops Agent 2.50.0, fluent-bit's tail input plugin causes **8-9x read amplification** on disk due to the `file_cache_advise` feature being enabled by default.
### Impact
- **Working:** Ops Agent 2.49.0 (fluent-bit v3.0.4) → ~0 KB/s disk reads
- **Broken:** Ops Agent 2.50.0+ (fluent-bit v3.1.6) → 4-5 MB/s sustained disk reads
- Disk utilization jumps from 0.3% to 84-90%
### Root Cause
Fluent-bit commit [8e0941e](https://github.com/fluent/fluent-bit/commit/8e0941e) introduced `file_cache_advise` (defaults to `true`) which calls `posix_fadvise(POSIX_FADV_DONTNEED)` after reading, evicting pages from kernel cache.
With multiple pipelines reading the same files + WAL buffering, data is read multiple times. Cache eviction forces every re-read to hit disk instead of being served from RAM.
### Reproduction
1. Deploy Ops Agent 2.50.0 on a VM with file-based logging receivers
2. Configure 2+ pipelines reading the same log files (common pattern)
3. Observe sustained high disk reads even when fluent-bit is caught up
Tested on:
- GCP VMs running Ubuntu 22.04
- `/data/logs/*/output.log` files rotating every 2-3 hours (2 GB each)
- Two pipelines: `default_pipeline` and `http_request_pipeline`
### Proposed Solution
Expose `file_cache_advise` as a user-configurable option in the Ops Agent YAML config, defaulting to `false` to match pre-2.50.0 behavior.
```yaml
logging:
receivers:
my-logs:
type: files
include_paths:
- /var/log/*.log
file_cache_advise: false # New option
```
### Workaround
Pin to Ops Agent 2.49.0:
```bash
sudo apt-mark hold google-cloud-ops-agent
sudo apt-get install google-cloud-ops-agent=2.49.0~ubuntu22.04
```
One server upgraded to 2.50.0 one downgraded to 2.49.0
Contributor guide
Assessment
This issue has not been assessed yet.