filebeat: stale harvester goroutine can re-ingest file after rotation when `harvester_limit` is reached
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
When `harvester_limit` is set and the number of monitored files exceeds the limit, a file can be harvested more than once due to stale goroutines waiting on the task group semaphore. This causes duplicate log ingestion.
PRs #48445 and https://github.com/elastic/beats/pull/49114 fixed the **goroutine leak** (OpWrite events piling up goroutines for files that already have a running harvester). However, there is still a scenario that causes re-ingestion that those fixes do not cover.
## Scenario
With `harvester_limit: 1` (for simplicity) and two files:
```
1. file-0.log is discovered → OpCreate → harvester starts, occupies the only slot
2. fileA.log is discovered → OpCreate → goroutine A blocks on semaphore
(no harvester registered in readers table yet — store.Get hasn't been called)
3. fileA.log grows → OpWrite → goroutine B blocks on semaphore
(PR #48445 adds a hasID check, but A hasn't registered yet, so B is spawned)
4. fileA.log is renamed to fileA.log.1 → OpRename
5. file-0.log harvester finishes → semaphore freed
6. Goroutine A runs → registers in readers, harvests fileA.log
7. Goroutine A finishes → removes itself from readers
8. Goroutine B runs → readers table is empty for this source → registers, harvests again
```
At step 3, the `hasID` / `reserve` check does not prevent goroutine B because goroutine A has not yet acquired the semaphore and therefore has not registered in the readers table.
## Impact
When the stale goroutine eventually runs after rotation:
- It opens the file at the **original path** (captured at event time), which now has different content (post-rotation)
- `openFile` detects `fi.Size() < stored_offset` → "File was truncated. Reading from offset 0"
- The file content is re-ingested, duplicating events that another harvester (for the new fingerprint) is also reading
## Related
- #48445
- https://github.com/elastic/beats/pull/49114
For confirmed bugs, please report:
- Version: 8.18 - but main uses the same architecture
- Operating System: All
- Discuss Forum URL: N/A
- Steps to Reproduce: see description
Contributor guide
Assessment
This issue has not been assessed yet.