elastic / elastic/beats

[filestream-registry] Filestream copytruncate Continue can discard pending cursor delta and regress resume offset

Open
#49,609 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Findings

### 1. Pending cursor delta is dropped during `Continue` state handoff (high)

**Location**
- `filebeat/input/filestream/internal/input-logfile/harvester.go:327-349`
- `filebeat/input/filestream/internal/input-logfile/store.go:782-799`
- `filebeat/input/filestream/internal/input-logfile/publish.go:99-109`
- `filebeat/input/filestream/internal/input-logfile/store.go:842-845`
- `filebeat/input/filestream/input.go:441-452`

**Evidence**
- `Continue` removes old state, then copies previous resource into next:
- `harvester.go:340`: `_ = hg.store.remove(prevID)`
- `harvester.go:348`: `previousResource.copyInto(nextResource)`
- `copyInto` keeps `activeCursorOperations` but clears pending cursor delta:
- `store.go:795`: `dst.activeCursorOperations = r.activeCursorOperations`
- `store.go:797-798`: `dst.pendingCursorValue = nil`, `dst.pendingUpdate = nil`
- Pending updates are created on publish:
- `publish.go:105`: `resource.pendingUpdate = updates`
- `publish.go:108`: `resource.activeCursorOperations++`
- Cursor read path uses pending state when active ops exist:
- `store.go:843-845`: if `activeCursorOperations != 0`, return `pendingCursor()`
- Harvester startup unpacks that cursor into initial file state:
- `input.go:447`: `c.Unpack(&state)`

**What is wrong**
During copytruncate rotation, `Continue` can transfer a resource where `activeCursorOperations > 0` but the associated pending delta is erased. That creates an inconsistent in-memory cursor state for the new resource.

**Why it matters**
Under output backpressure/delayed ACKs, resume offset for the rotated file can regress to the last fully persisted cursor (or zero if no persisted cursor exists), causing re-ingestion of already published bytes after restart/harvester reopen.

**Realistic reproduction scenario**
1. Configure filestream with copytruncate rotation.
2. Generate log traffic while slowing/blocking output ACKs.
3. Trigger rotation so `onRotatedFile` calls `hg.Continue(...)` (`copytruncate_prospector.go:343-347`).
4. Restart Filebeat during/after this window.
5. Observe resumed offset behind expected point for rotated file (duplicate ingestion).

**Suggested fix direction**
Make handoff state internally consistent in `copyInto` (and equivalent handoff helpers):
- either copy pending cursor payload (`pendingUpdate`/`pendingCursorValue`) when copying `activeCursorOperations`,
- or zero out `activeCursorOperations` and materialize a consistent cursor snapshot before transfer.

## Scenarios investigated and found safe
- Identifier matching migration guards avoid wrong-file key migration (`filebeat/input/filestream/prospector.go:240-283`).
- Cleanup path checks pending owners before hard removal (`filebeat/input/filestream/internal/input-logfile/clean.go:119-136`).
- ACK wiring (`newInputACKHandler` -> update channel -> `updateOp.Execute`) is present and ordered (`filebeat/input/filestream/internal/input-logfile/input.go:104-129`, `update_writer.go:96-100`).

## Priority ranking
1. **High / production-likely:** copytruncate + output backpressure + restart window (can produce duplicate ingestion bursts).
2. **Lower:** startup-only identifier migration windows (`UpdateIdentifiers`/`TakeOver`) are less likely to have in-flight ops and were not confirmed as a realistic trigger in normal runtime.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Filestream Registry and State Machine](https://github.com/elastic/beats/actions/runs/23450914112)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Mar 30, 2026, 5:47 PM UTC

Contributor guide

Open the contributing guide

Research direction

Trace the Continue handoff in filebeat/input/filestream/internal/input-logfile/harvester.go alongside copyInto in store.go, then follow pending updates through publish.go and cursor unpacking in filebeat/input/filestream/input.go. Use the copytruncate_prospector.go rotation path and the described delayed-ACK restart scenario as the validation entry point; done means the transferred cursor state remains internally consistent and the rotated file resumes without duplicate ingestion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.