GoogleCloudPlatform / GoogleCloudPlatform/ops-agent

Ops Agent 2.50.0+: Excessive disk reads due to file_cache_advise default

Open
#2,307 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
204
Forks
87
Avg merge
21h 54m
Merged PRs (30d)
23

Description

## Problem

Starting with Ops Agent 2.50.0, fluent-bit's tail input plugin causes **8-9x read amplification** on disk due to the `file_cache_advise` feature being enabled by default.

### Impact

- **Working:** Ops Agent 2.49.0 (fluent-bit v3.0.4) → ~0 KB/s disk reads
- **Broken:** Ops Agent 2.50.0+ (fluent-bit v3.1.6) → 4-5 MB/s sustained disk reads
- Disk utilization jumps from 0.3% to 84-90%

### Root Cause

Fluent-bit commit [8e0941e](https://github.com/fluent/fluent-bit/commit/8e0941e) introduced `file_cache_advise` (defaults to `true`) which calls `posix_fadvise(POSIX_FADV_DONTNEED)` after reading, evicting pages from kernel cache.

With multiple pipelines reading the same files + WAL buffering, data is read multiple times. Cache eviction forces every re-read to hit disk instead of being served from RAM.

### Reproduction

1. Deploy Ops Agent 2.50.0 on a VM with file-based logging receivers
2. Configure 2+ pipelines reading the same log files (common pattern)
3. Observe sustained high disk reads even when fluent-bit is caught up

Tested on:
- GCP VMs running Ubuntu 22.04
- `/data/logs/*/output.log` files rotating every 2-3 hours (2 GB each)
- Two pipelines: `default_pipeline` and `http_request_pipeline`

### Proposed Solution

Expose `file_cache_advise` as a user-configurable option in the Ops Agent YAML config, defaulting to `false` to match pre-2.50.0 behavior.

```yaml
logging:
receivers:
my-logs:
type: files
include_paths:
- /var/log/*.log
file_cache_advise: false # New option
```

### Workaround

Pin to Ops Agent 2.49.0:
```bash
sudo apt-mark hold google-cloud-ops-agent
sudo apt-get install google-cloud-ops-agent=2.49.0~ubuntu22.04
```

One server upgraded to 2.50.0 one downgraded to 2.49.0
Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.