elastic / elastic/integrations

[system] Security data stream: `include_xml` is unconditionally hardcoded (regression) — no longer gated by "Preserve original event"

Open
#20,940 2 comments 0 reactions 0 assignees View on GitHub
Integration:system Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## Summary

In the System integration's **`security`** data stream, `include_xml: true` is emitted **unconditionally** in the agent stream template. This appears to be an **accidental regression**: it used to be gated behind the `preserve_original_event` toggle (as it still is for the `application` and `system` data streams), but the gate was lost during an unrelated change and then cemented by a follow-up syntax bugfix.

As a result, on every Security event the agent always renders the full raw event XML (`event.original`) regardless of the user's "Preserve original event" setting. The Fleet final pipeline still drops `event.original` at ingest when the tag is absent, so **index storage** is unaffected — but the **agent-side XML rendering cost and the network egress** of that XML are always incurred and cannot be turned off via the UI. On high-volume Security channels (e.g. Domain Controllers, Exchange servers producing large numbers of 4624/4634/4672), this is a measurable, non-optional CPU/RAM/bandwidth cost.

## Affected file

`packages/system/data_stream/security/agent/stream/winlog.yml.hbs`

Current (unconditional):

```handlebars
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
include_xml: true
processors:
- decode_xml:
...
```

Compare `packages/system/data_stream/application/agent/stream/winlog.yml.hbs`, which correctly gates it:

```handlebars
{{#if preserve_original_event}}
include_xml: true
{{/if}}
```

## Regression provenance

| PR | Date | Security template state |
|----|------|-------------------------|
| #2542 | 2022-01 | `{{#if preserve_original_event}} include_xml: true {{/if}}` — **gated** (same as application) |
| #9426 | 2024-04 | still gated |
| #9045 | 2024-05 | still gated |
| **#13828** | 2025-07-24 (am) | tags block restructured; `{{#if preserve_original_event}}` opener around `include_xml` **dropped**, leaving a **dangling `{{/if}}`** (broken handlebars) |
| **#14674** | 2025-07-24 (pm) | *"Fix condition in handlebar file"* — removed the dangling `{{/if}}`, leaving `include_xml: true` **unconditional**. Gate not restored. |

Only the `security` data stream was touched by #13828; `application` and `system` retained the correct gating, which is why they diverge today. Neither #13828 nor #14674 mentions `include_xml` or any intent to always collect raw XML for security events, so this looks unintentional.

## Impact

For the `security` data stream, with **"Preserve original event" = off**:

- Agent still uses the XML renderer (`NewXMLRenderer`) and populates `event.original` for every event — **CPU cost is always paid**.
- `event.original` is transmitted to Elasticsearch, then removed by the Fleet final pipeline (`.fleet_final_pipeline-1`) because the `preserve_original_event` tag is absent — **network egress is always paid; only index storage is saved**.
- The UI toggle therefore has **no effect** on agent-side CPU/RAM or network for this data stream, contrary to its behavior on `application`/`system` and contrary to user expectation.

## Not related: `decode_xml` processors

The two `decode_xml` processors in the same template are **independent** of `include_xml`. They parse `winlog.event_data.TaskContent` / `TaskContentNew` (embedded scheduled-task definition XML) for events 4698–4702, feeding `scheduled_task.yml` → `winlog.scheduled_task.*`. Those fields exist in `winlog.event_data.*` regardless of `include_xml` (both renderers populate event data), so `decode_xml` must remain **unconditional**. This proposal does **not** touch them.

## Proposed fix

Restore the gate around `include_xml` only, matching the `application`/`system` templates, and leave the `decode_xml` processors unconditional:

```handlebars
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
{{#if preserve_original_event}}
include_xml: true
{{/if}}
processors:
- decode_xml:
when:
and:
- has_fields:
- winlog.event_data.TaskContent
- equals:
winlog.provider_name: Microsoft-Windows-Security-Auditing
field: winlog.event_data.TaskContent
target_field: winlog._tmp.scheduled_task.task_content
to_lower: true
ignore_missing: true
ignore_failure: true
- decode_xml:
when:
and:
- has_fields:
- winlog.event_data.TaskContentNew
- equals:
winlog.provider_name: Microsoft-Windows-Security-Auditing
field: winlog.event_data.TaskContentNew
target_field: winlog._tmp.scheduled_task.task_content_new
to_lower: true
ignore_missing: true
ignore_failure: true
```

Note: for `forwarded` inputs the agent already forces the XML renderer (`c.IncludeXML || l.isForwarded()` in `winlogbeat/eventlog/wineventlog.go`), so forwarded collection is unaffected by re-gating `include_xml`.

## Validation notes

- With `include_xml` off, the agent switches from `NewXMLRenderer` (XML-parse) to `NewRenderer` (EvtRender). Both populate the same `winlog.*` structured fields, but pipeline tests should confirm field parity for the Security event set (especially UserData-based events and any legacy/manifest-less providers).
- Confirm `winlog.scheduled_task.*` still populates for 4698–4702 with the toggle off (it should, since `TaskContent` is normal event data).
- Confirm `event.original` is present only when the toggle is on.

Contributor guide

Open the contributing guide

Research direction

Start with packages/system/data_stream/security/agent/stream/winlog.yml.hbs and compare its include_xml handling with the application and system templates. Run the relevant pipeline tests, checking that event.original follows the preserve_original_event toggle, scheduled-task fields still populate when it is off, and forwarded inputs remain unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
handlebars
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.