windows_event_log: source does not re-subscribe on ERROR_EVT_QUERY_RESULT_STALE (0x80073AA3) and gets stuck in a pull loop on high-volume channels

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Summary

On a high-volume Windows channel — the Security log on a Domain Controller — the windows_event_log source stops ingesting after the log rolls over and never recovers on its own. A pull returns ERROR_EVT_QUERY_RESULT_STALE (0x80073AA3); the source then logs "Recoverable pull error, backing off." and retries the same, now-invalid subscription with exponential backoff (100 ms doubling, capped at 5 s) — one failed pull every 5 s, indefinitely. It resumes only after a process restart. On an unattended fleet this presents as recurring multi-hour gaps in Security data.

This is specific to ERROR_EVT_QUERY_RESULT_STALE. The sibling error ERROR_EVT_QUERY_RESULT_INVALID_POSITION is already handled correctly (it re-subscribes), which is what makes the fix small and low-risk.

The same error, two code paths — one recovers, one does not

The same STALE error is handled in two places with opposite outcomes, and the Security channel alone shows both in one 0.57.0 run (verbatim logs in Debug Output):

• At SUBSCRIBE (process start) → RECOVERS. The open path retries without the bookmark and succeeds ("Strict bookmark subscribe failed, retrying without bookmark" → "Pull-mode subscription created successfully", 07:26:51). The identical recovery also fired on the Directory Service channel at the same startup.
• During PULL (~2h later, while reading) → DOES NOT RECOVER. At 09:36:09 the busy Security log rolled over past the open subscription; the pull path hit STALE and never re-subscribed — it backed off and retried the same dead subscription, still looping 17 min later (09:53:52). 436 STALE errors; ingestion stopped until restart.

Same channel, same error, opposite results → the problem is the code path, not the channel.

Why only the Security channel stalls

Every channel uses the subscribe path at start and recovers. The pull-path STALE is only triggered on a channel busy enough to roll its log over while Vector is mid-read — Security, by far the highest-volume channel here. Quiet channels (Directory Service, etc.) never reach it. The failure tracks event volume, consistent with a log-rollover trigger.

Root cause (v0.57.0 source)

In src/sources/windows_event_log/subscription.rs, inside the EvtNext drain loop:

• ERROR_EVT_QUERY_RESULT_INVALID_POSITION → calls resubscribe_channel() (subscription.rs:584-614)
• ERROR_EVT_QUERY_RESULT_STALE → only channel_drained = true; break;, no re-subscription (subscription.rs:574-583)

resubscribe_channel() (subscription.rs:806-895) already does the right thing: EvtClose the stale handle, EvtSubscribe again from the bookmark, and if the bookmark is itself stale, fall back to a bookmark-less subscription. The STALE branch never calls it. After the break, the pull loop in mod.rs (loop at mod.rs:389) backs off (100 ms → cap 5 s) and calls EvtNext on the same stale handle, which keeps returning STALE. The "speculative timeout" idle-cycle pull (mod.rs:496-524, from the lost-wakeup fix #25194/#25195) reuses the same stale handle too, so it cannot rescue a STALE subscription.

0.57.0's subscribe-time fallback (subscription.rs:269-314) runs only when a subscription is opened (start/restart, or inside resubscribe_channel) — never from the pull path. So STALE-while-reading recovers only via a restart. Same handling on 0.56.0, 0.57.0 and master.

What Microsoft says

Per WinError.h, ERROR_EVT_QUERY_RESULT_STALE (Win32 15011 / HRESULT 0x80073AA3): "The query result is not valid. This may be due to the log being cleared or rolling over after the query result was created. Release the query result object and reissue the query." (https://learn.microsoft.com/en-us/windows/win32/wes/windows-event-log-error-constants). Under EvtSubscribeStrict the same error means "event records are missing" (EVT_SUBSCRIBE_CALLBACK). Microsoft's prescribed recovery is exactly release + reissue (= re-subscribe) — which the pull path fails to do.

Expected behavior

On STALE, release the stale result and re-subscribe (as already done for INVALID_POSITION), falling back to oldest available if the bookmark is gone — so the source self-recovers at runtime, with no restart and no operator intervention.

Actual behavior

No re-subscription on STALE; the source retries the invalidated handle forever and the channel stops ingesting until Vector is restarted.

Suggested fix

In the STALE arm (subscription.rs:574-583), call the existing resubscribe_channel() — exactly as the INVALID_POSITION arm (subscription.rs:584-614) already does — instead of breaking. A few lines, reusing code that already handles the bookmark-stale fallback; matches Microsoft's documented remedy.

Configuration

sources:
  wineventlog_security:
    type: windows_event_log
    channels: ["Security"]
    read_existing_events: false
    # batch_size (100), event_timeout_ms (5000), events_per_second (0) all left at defaults

Version

vector 0.57.0 (x86_64-pc-windows-msvc 8832452 2026-07-14) — also reproduced on 0.56.0; same pull-loop handling on master.

Debug Output

Relevant internal_logs lines (verbatim; only the host value redacted). Same 0.57.0 run.

WORKS — STALE at SUBSCRIBE recovers (Security, process start):
{"channel":"Security","error":"The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created. (0x80073AA3)","fallback_flags":"0x1","host":"REDACTED","message":"Strict bookmark subscribe failed, retrying without bookmark. Potential re-delivery of events.","metadata":{"kind":"event","level":"WARN","module_path":"vector::sources::windows_event_log::subscription","target":"vector::sources::windows_event_log::subscription"},"pid":6156,"source_type":"internal_logs","timestamp":"2026-08-14T07:26:51.364098600Z","vector":{"component_id":"wineventlog_security","component_kind":"source","component_type":"windows_event_log"}}
{"channel":"Security","host":"REDACTED","message":"Pull-mode subscription created successfully.","metadata":{"kind":"event","level":"INFO","module_path":"vector::sources::windows_event_log::subscription","target":"vector::sources::windows_event_log::subscription"},"pid":6156,"source_type":"internal_logs","timestamp":"2026-08-14T07:26:51.364286200Z","vector":{"component_id":"wineventlog_security","component_kind":"source","component_type":"windows_event_log"}}

Same subscribe-time recovery on Directory Service (not channel-specific):
{"channel":"Directory Service","error":"The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created. (0x80073AA3)","fallback_flags":"0x2","host":"REDACTED","message":"Strict bookmark subscribe failed, retrying without bookmark. Potential re-delivery of events.","metadata":{"kind":"event","level":"WARN","module_path":"vector::sources::windows_event_log::subscription","target":"vector::sources::windows_event_log::subscription"},"pid":6156,"source_type":"internal_logs","timestamp":"2026-08-14T07:26:51.361743200Z","vector":{"component_id":"wineventlog_directory_service","component_kind":"source","component_type":"windows_event_log"}}
{"channel":"Directory Service","host":"REDACTED","message":"Pull-mode subscription created successfully.","metadata":{"kind":"event","level":"INFO","module_path":"vector::sources::windows_event_log::subscription","target":"vector::sources::windows_event_log::subscription"},"pid":6156,"source_type":"internal_logs","timestamp":"2026-08-14T07:26:51.362377900Z","vector":{"component_id":"wineventlog_directory_service","component_kind":"source","component_type":"windows_event_log"}}

DOES NOT WORK — STALE during PULL never re-subscribes (Security); onset, then still looping 17 min later at the 5 s cap:
{"channel":"all","error":"Failed to pull events from channel 'Security': The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created. (0x80073AA3)","error_code":"query_failed","error_type":"request_failed","host":"REDACTED","internal_log_rate_limit":true,"message":"Failed to query Windows Event Log.","metadata":{"kind":"event","level":"ERROR","module_path":"vector::internal_events::windows_event_log","target":"vector::internal_events::windows_event_log"},"pid":4720,"query":"None","source_type":"internal_logs","stage":"receiving","timestamp":"2026-08-14T09:36:09.680256400Z","vector":{"component_id":"wineventlog_security","component_kind":"source","component_type":"windows_event_log"}}
{"backoff_ms":5000,"error":"Failed to pull events from channel 'Security': The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created. (0x80073AA3)","host":"REDACTED","message":"Recoverable pull error, backing off.","metadata":{"kind":"event","level":"WARN","module_path":"vector::sources::windows_event_log","target":"vector::sources::windows_event_log"},"pid":4720,"source_type":"internal_logs","timestamp":"2026-08-14T09:53:52.434609200Z","vector":{"component_id":"wineventlog_security","component_kind":"source","component_type":"windows_event_log"}}

Separate, self-recovered lost wakeup on Directory Service at the same instant (not a STALE):
{"event_count":1,"host":"REDACTED","message":"Speculative timeout pull recovered events; possible lost wakeup detected.","metadata":{"kind":"event","level":"WARN","module_path":"vector::sources::windows_event_log","target":"vector::sources::windows_event_log"},"pid":4720,"source_type":"internal_logs","timestamp":"2026-08-14T09:36:09.621930100Z","vector":{"component_id":"wineventlog_directory_service","component_kind":"source","component_type":"windows_event_log"}}

Example Data

N/A — subscription-recovery bug, not a data/format issue.

Additional Context

• Runs as a Windows service on a Domain Controller; ~9 windows_event_log channels as independent source instances (Application, System, Security, Directory Service, DFS Replication, DNS Server/Audit, among others) plus a file source and a scheduled exec source, all feeding one sink. No Kubernetes, no unusual CLI flags/env vars.
• Timeline: Security collected normally ~2h09m (07:27 → 09:36:09 UTC), then STALE; 436 STALE pull errors; ingestion stopped until restart. Same on 0.56.0 (~36,000 STALE errors over ~2 days, multi-hour Security gaps) — not specific to one build.

References

Same defect already fixed in Elastic Beats (shared winlog code) — established fix pattern:

• elastic/beats#32168 — "filebeat: winlog input stops when ERROR_EVT_QUERY_RESULT_STALE 15011 is encountered" (closed). STALE was not in the retryable-errors set; the fix added it, citing Microsoft's "release the query result object and reissue the query".
• elastic/beats#45750 — "Winlog Beat does not recover from recoverable error when reading from Windows Event Log" (closed). Same STALE message; repro forces it with a small log max size so the log rolls over before events are read.
• vectordotdev/vector#25194 / #25195 — related lost-wakeup fix in the same source (different code path; does not cover this STALE pull case).

Microsoft docs:

• WinError.h constants: https://learn.microsoft.com/en-us/windows/win32/wes/windows-event-log-error-constants
• EVT_SUBSCRIBE_CALLBACK: https://learn.microsoft.com/en-us/windows/win32/api/winevt/nc-winevt-evt_subscribe_callback
• EvtSubscribe: https://learn.microsoft.com/en-us/windows/win32/api/winevt/nf-winevt-evtsubscribe
• Subscribing to Events: https://learn.microsoft.com/en-us/windows/win32/wes/subscribing-to-events

Start in src/sources/windows_event_log/subscription.rs at the EvtNext drain loop and compare the STALE branch with the existing INVALID_POSITION recovery. Read resubscribe_channel() and the pull loop in src/sources/windows_event_log/mod.rs. Done means a stale pull subscription is recreated with bookmark fallback and the Windows event source resumes without a restart.