windows_event_log: re-subscription drops records when the strict-bookmark resubscribe fails with ERROR_NOT_FOUND and falls back to future-events (follow-up to #26118)

This is the follow-up defect flagged during review of #26118 (the reviewer's "Preserve queued records when bookmark fallback is required" point). It only becomes reachable once re-subscription actually runs — which #26118 is what enables — so it cannot be reproduced on main as-is, but it is a real correctness problem in the re-subscription path itself. Filing separately as requested there.

Summary

On a domain-joined Windows Server with read_existing_events = false, the windows_event_log source recovers dead subscriptions correctly (no more silent stall — that part works), but resubscribe_channel fails its strict-bookmark EvtSubscribe on every recovery cycle with ERROR_NOT_FOUND (0x80070490), falls back to EvtSubscribeToFutureEvents, and records written between/around the rebuild are silently dropped.

read_existing_events = false is effectively lossy for event-log delivery on this host, even though the source looks healthy and busy.

Environment

• Vector built from the #26118 branch (0.58.0, x86_64-pc-windows-msvc).
• Windows Server 2025, domain-joined, IIS host.
• Config: six windows_event_log sources (Application/System/Security + two DHCP channels + a custom channel), read_existing_events = false, GELF/UDP sink, plus internal_logs → file for diagnostics. A parallel shipper (nxlog) runs on the same host, which is how the loss was isolated.

Notably, the same config on a client VM did not exhibit this — there the strict-bookmark resubscribe succeeded and delivery worked. The server is where the strict resubscribe consistently returns ERROR_NOT_FOUND.

Evidence (measured, ~5 min after start)

internal_logs, default level:

{"channel":"Security","error":"Element not found. (0x80070490)","fallback_flags":"0x1",
 "message":"Strict bookmark resubscribe failed, retrying without bookmark. Potential re-delivery of events.",
 "metadata":{"level":"WARN"},"source_type":"internal_logs", ...}

Controlled delivery check: two events written to the Application log with eventcreate (Warning + Error, i.e. within the source's level filter) appeared in the Windows Application log but were not delivered by Vector to the sink (verified via a distinguishing tag; only the parallel shipper's copies arrived). A file-source line written at the same time was delivered — so the sink/pipeline is fine; the loss is at the event-log source, in the fallback.

Why records are lost

When the strict-bookmark EvtSubscribe fails, resubscribe_channel falls back to EvtSubscribeToFutureEvents (because read_existing_events = false). That resets the position to "from now", so any record produced between the last consumed record and the moment the replacement subscription is created is skipped. With the resubscribe cycling every few seconds, there is a continuous stream of these gaps.

The ERROR_NOT_FOUND on the strict-bookmark subscribe itself is the deeper question — it recurs on every cycle on this host, which both drives the churn and forces the lossy fallback.

Suggested directions

• Retry a transient strict-bookmark failure without discarding the bookmark, rather than immediately dropping to future-events.
• When the bookmark is genuinely stale, resume from the oldest available record in the (possibly rotated) log instead of jumping to future-events, so the gap is closed rather than skipped.
• Investigate why EvtSubscribe with the resumed bookmark returns ERROR_NOT_FOUND (0x80070490) so consistently on Windows Server (vs. a client) — that is what makes the fallback the norm rather than the exception here, and may point at how/when the bookmark handle is built during recovery.

Offer

I have a reproducing environment (the domain-joined Server where this happens consistently) and am happy to test a candidate build against it.

Start by tracing the windows_event_log resubscribe_channel path enabled by #26118, focusing on strict-bookmark failure and the EvtSubscribeToFutureEvents fallback. Reproduce on the described domain-joined Windows Server with read_existing_events = false and inspect internal_logs for ERROR_NOT_FOUND. Done means recovery no longer silently drops records when strict resubscription fails, with behavior validated against the supplied eventcreate check.