elastic / elastic/integrations
[system] Security data stream: `include_xml` is unconditionally hardcoded (regression) — no longer gated by "Preserve original event"
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Summary
In the System integration's **`security`** data stream, `include_xml: true` is emitted **unconditionally** in the agent stream template. This appears to be an **accidental regression**: it used to be gated behind the `preserve_original_event` toggle (as it still is for the `application` and `system` data streams), but the gate was lost during an unrelated change and then cemented by a follow-up syntax bugfix.
As a result, on every Security event the agent always renders the full raw event XML (`event.original`) regardless of the user's "Preserve original event" setting. The Fleet final pipeline still drops `event.original` at ingest when the tag is absent, so **index storage** is unaffected — but the **agent-side XML rendering cost and the network egress** of that XML are always incurred and cannot be turned off via the UI. On high-volume Security channels (e.g. Domain Controllers, Exchange servers producing large numbers of 4624/4634/4672), this is a measurable, non-optional CPU/RAM/bandwidth cost.
## Affected file
`packages/system/data_stream/security/agent/stream/winlog.yml.hbs`
Current (unconditional):
```handlebars
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
include_xml: true
processors:
- decode_xml:
...
```
Compare `packages/system/data_stream/application/agent/stream/winlog.yml.hbs`, which correctly gates it:
```handlebars
{{#if preserve_original_event}}
include_xml: true
{{/if}}
```
## Regression provenance
| PR | Date | Security template state |
|----|------|-------------------------|
| #2542 | 2022-01 | `{{#if preserve_original_event}} include_xml: true {{/if}}` — **gated** (same as application) |
| #9426 | 2024-04 | still gated |
| #9045 | 2024-05 | still gated |
| **#13828** | 2025-07-24 (am) | tags block restructured; `{{#if preserve_original_event}}` opener around `include_xml` **dropped**, leaving a **dangling `{{/if}}`** (broken handlebars) |
| **#14674** | 2025-07-24 (pm) | *"Fix condition in handlebar file"* — removed the dangling `{{/if}}`, leaving `include_xml: true` **unconditional**. Gate not restored. |
Only the `security` data stream was touched by #13828; `application` and `system` retained the correct gating, which is why they diverge today. Neither #13828 nor #14674 mentions `include_xml` or any intent to always collect raw XML for security events, so this looks unintentional.
## Impact
For the `security` data stream, with **"Preserve original event" = off**:
- Agent still uses the XML renderer (`NewXMLRenderer`) and populates `event.original` for every event — **CPU cost is always paid**.
- `event.original` is transmitted to Elasticsearch, then removed by the Fleet final pipeline (`.fleet_final_pipeline-1`) because the `preserve_original_event` tag is absent — **network egress is always paid; only index storage is saved**.
- The UI toggle therefore has **no effect** on agent-side CPU/RAM or network for this data stream, contrary to its behavior on `application`/`system` and contrary to user expectation.
## Not related: `decode_xml` processors
The two `decode_xml` processors in the same template are **independent** of `include_xml`. They parse `winlog.event_data.TaskContent` / `TaskContentNew` (embedded scheduled-task definition XML) for events 4698–4702, feeding `scheduled_task.yml` → `winlog.scheduled_task.*`. Those fields exist in `winlog.event_data.*` regardless of `include_xml` (both renderers populate event data), so `decode_xml` must remain **unconditional**. This proposal does **not** touch them.
## Proposed fix
Restore the gate around `include_xml` only, matching the `application`/`system` templates, and leave the `decode_xml` processors unconditional:
```handlebars
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
{{#if preserve_original_event}}
include_xml: true
{{/if}}
processors:
- decode_xml:
when:
and:
- has_fields:
- winlog.event_data.TaskContent
- equals:
winlog.provider_name: Microsoft-Windows-Security-Auditing
field: winlog.event_data.TaskContent
target_field: winlog._tmp.scheduled_task.task_content
to_lower: true
ignore_missing: true
ignore_failure: true
- decode_xml:
when:
and:
- has_fields:
- winlog.event_data.TaskContentNew
- equals:
winlog.provider_name: Microsoft-Windows-Security-Auditing
field: winlog.event_data.TaskContentNew
target_field: winlog._tmp.scheduled_task.task_content_new
to_lower: true
ignore_missing: true
ignore_failure: true
```
Note: for `forwarded` inputs the agent already forces the XML renderer (`c.IncludeXML || l.isForwarded()` in `winlogbeat/eventlog/wineventlog.go`), so forwarded collection is unaffected by re-gating `include_xml`.
## Validation notes
- With `include_xml` off, the agent switches from `NewXMLRenderer` (XML-parse) to `NewRenderer` (EvtRender). Both populate the same `winlog.*` structured fields, but pipeline tests should confirm field parity for the Security event set (especially UserData-based events and any legacy/manifest-less providers).
- Confirm `winlog.scheduled_task.*` still populates for 4698–4702 with the toggle off (it should, since `TaskContent` is normal event data).
- Confirm `event.original` is present only when the toggle is on.
Contributor guide
Research direction
Start with packages/system/data_stream/security/agent/stream/winlog.yml.hbs and compare its include_xml handling with the application and system templates. Run the relevant pipeline tests, checking that event.original follows the preserve_original_event toggle, scheduled-task fields still populate when it is off, and forwarded inputs remain unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- handlebars
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100