elastic / elastic/beats

filebeat: stale harvester goroutine can re-ingest file after rotation when `harvester_limit` is reached

Open
#50,002 4 comments 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

When `harvester_limit` is set and the number of monitored files exceeds the limit, a file can be harvested more than once due to stale goroutines waiting on the task group semaphore. This causes duplicate log ingestion.

PRs #48445 and https://github.com/elastic/beats/pull/49114 fixed the **goroutine leak** (OpWrite events piling up goroutines for files that already have a running harvester). However, there is still a scenario that causes re-ingestion that those fixes do not cover.

## Scenario

With `harvester_limit: 1` (for simplicity) and two files:

```
1. file-0.log is discovered → OpCreate → harvester starts, occupies the only slot
2. fileA.log is discovered → OpCreate → goroutine A blocks on semaphore
(no harvester registered in readers table yet — store.Get hasn't been called)
3. fileA.log grows → OpWrite → goroutine B blocks on semaphore
(PR #48445 adds a hasID check, but A hasn't registered yet, so B is spawned)
4. fileA.log is renamed to fileA.log.1 → OpRename
5. file-0.log harvester finishes → semaphore freed
6. Goroutine A runs → registers in readers, harvests fileA.log
7. Goroutine A finishes → removes itself from readers
8. Goroutine B runs → readers table is empty for this source → registers, harvests again
```

At step 3, the `hasID` / `reserve` check does not prevent goroutine B because goroutine A has not yet acquired the semaphore and therefore has not registered in the readers table.

## Impact

When the stale goroutine eventually runs after rotation:
- It opens the file at the **original path** (captured at event time), which now has different content (post-rotation)
- `openFile` detects `fi.Size() < stored_offset` → "File was truncated. Reading from offset 0"
- The file content is re-ingested, duplicating events that another harvester (for the new fingerprint) is also reading

## Related

- #48445
- https://github.com/elastic/beats/pull/49114

For confirmed bugs, please report:
- Version: 8.18 - but main uses the same architecture
- Operating System: All
- Discuss Forum URL: N/A
- Steps to Reproduce: see description

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.