[filestream-registry] Filestream copytruncate Continue can discard pending cursor delta and regress resume offset
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Findings
### 1. Pending cursor delta is dropped during `Continue` state handoff (high)
**Location**
- `filebeat/input/filestream/internal/input-logfile/harvester.go:327-349`
- `filebeat/input/filestream/internal/input-logfile/store.go:782-799`
- `filebeat/input/filestream/internal/input-logfile/publish.go:99-109`
- `filebeat/input/filestream/internal/input-logfile/store.go:842-845`
- `filebeat/input/filestream/input.go:441-452`
**Evidence**
- `Continue` removes old state, then copies previous resource into next:
- `harvester.go:340`: `_ = hg.store.remove(prevID)`
- `harvester.go:348`: `previousResource.copyInto(nextResource)`
- `copyInto` keeps `activeCursorOperations` but clears pending cursor delta:
- `store.go:795`: `dst.activeCursorOperations = r.activeCursorOperations`
- `store.go:797-798`: `dst.pendingCursorValue = nil`, `dst.pendingUpdate = nil`
- Pending updates are created on publish:
- `publish.go:105`: `resource.pendingUpdate = updates`
- `publish.go:108`: `resource.activeCursorOperations++`
- Cursor read path uses pending state when active ops exist:
- `store.go:843-845`: if `activeCursorOperations != 0`, return `pendingCursor()`
- Harvester startup unpacks that cursor into initial file state:
- `input.go:447`: `c.Unpack(&state)`
**What is wrong**
During copytruncate rotation, `Continue` can transfer a resource where `activeCursorOperations > 0` but the associated pending delta is erased. That creates an inconsistent in-memory cursor state for the new resource.
**Why it matters**
Under output backpressure/delayed ACKs, resume offset for the rotated file can regress to the last fully persisted cursor (or zero if no persisted cursor exists), causing re-ingestion of already published bytes after restart/harvester reopen.
**Realistic reproduction scenario**
1. Configure filestream with copytruncate rotation.
2. Generate log traffic while slowing/blocking output ACKs.
3. Trigger rotation so `onRotatedFile` calls `hg.Continue(...)` (`copytruncate_prospector.go:343-347`).
4. Restart Filebeat during/after this window.
5. Observe resumed offset behind expected point for rotated file (duplicate ingestion).
**Suggested fix direction**
Make handoff state internally consistent in `copyInto` (and equivalent handoff helpers):
- either copy pending cursor payload (`pendingUpdate`/`pendingCursorValue`) when copying `activeCursorOperations`,
- or zero out `activeCursorOperations` and materialize a consistent cursor snapshot before transfer.
## Scenarios investigated and found safe
- Identifier matching migration guards avoid wrong-file key migration (`filebeat/input/filestream/prospector.go:240-283`).
- Cleanup path checks pending owners before hard removal (`filebeat/input/filestream/internal/input-logfile/clean.go:119-136`).
- ACK wiring (`newInputACKHandler` -> update channel -> `updateOp.Execute`) is present and ordered (`filebeat/input/filestream/internal/input-logfile/input.go:104-129`, `update_writer.go:96-100`).
## Priority ranking
1. **High / production-likely:** copytruncate + output backpressure + restart window (can produce duplicate ingestion bursts).
2. **Lower:** startup-only identifier migration windows (`UpdateIdentifiers`/`TakeOver`) are less likely to have in-flight ops and were not confirmed as a realistic trigger in normal runtime.
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Filestream Registry and State Machine](https://github.com/elastic/beats/actions/runs/23450914112)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Mar 30, 2026, 5:47 PM UTC
Contributor guide
Research direction
Trace the Continue handoff in filebeat/input/filestream/internal/input-logfile/harvester.go alongside copyInto in store.go, then follow pending updates through publish.go and cursor unpacking in filebeat/input/filestream/input.go. Use the copytruncate_prospector.go rotation path and the described delayed-ACK restart scenario as the validation entry point; done means the transferred cursor state remains internally consistent and the rotated file resumes without duplicate ingestion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100