Azure / Azure/AzureMonitor

AMA Windows Event Logs: Ambiguous behavior when both `Microsoft-Event` and `Microsoft-WindowsEvent` streams are configured

Open
#12 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
No language data
Stars
1
Forks
8
PR merge metrics
No merged PRs in 30d

Description

## Description

We observed inconsistent ingestion behavior when a single `windows_event_log` data source is configured with both:

```hcl
streams = [
"Microsoft-WindowsEvent",
"Microsoft-Event"
]
```

and deployed using the Terraform AzureRM provider.

The expected behavior is unclear:

- Are both streams expected to be used?
- Is one stream preferred over the other?
- Is the behavior deterministic?
- Which table should events ultimately land in (`Event` vs `WindowsEvent`)?

## Environment

### DCR Configuration

```hcl
data_sources {
windows_event_log {
streams = [
"Microsoft-WindowsEvent",
"Microsoft-Event"
]

x_path_queries = [
"Application!*[System[(Level=1 or Level=2)]]",
"System!*[System[(Level=1 or Level=2)]]"
]
}
}
```

### Infrastructure

- Azure Virtual Machines
- Azure Monitor Windows Agent (AMA)
- Log Analytics Workspace
- Data Collection Rules (DCR)
- DCR Associations managed via Terraform

## Observed Behavior

Initially, we noticed that some VMs were writing Windows event data to the **Event** table, while others were writing to the **WindowsEvent** table.

After further investigation using a workspace with a longer retention period, the behavior appears more complex:

- Some VMs historically wrote to `WindowsEvent` and later switched to `Event`.
- Some VMs have always written to `Event`.
- Some VMs have always written to `WindowsEvent`.
- The behavior is not consistent across all machines despite identical configuration.

## Investigation Performed

### Verified Identical Configuration

The following were verified across affected VMs:

- Same DCR
- Same DCR associations
- Same Log Analytics workspace
- Same VM type (`Microsoft.Compute/virtualMachines`)
- Same Terraform configuration
- Same current AMA version
- Same `configchunks` content under:

```text
C:\WindowsAzure\Resources\AMADataStore.*\mcs\configchunks
```

### AMA Internal Streams

All inspected VMs contained the following stream definitions:

```json
{
"stream": "SECURITY_WEF_EVENT_BLOB",
"solution": "SecurityInsights"
}
```
and
```json
{
"stream": "GENERIC_EVENT_BLOB",
"solution": "LogManagement"
}
```

The effective configuration files were identical between VMs that ingest into Event and VMs that ingest into WindowsEvent.

### AMA Upgrade Correlation

We found indications that a number of machines switched from `WindowsEvent` to `Event` around the time AMA was upgraded from approximately:

```text
1.39.x -> 1.42.x
```

However, not all machines behaved the same way:
- Not all VMs switched.
- Some VMs remained on `WindowsEvent`.
- Some VMs remained on `Event`.

## Questions

### Supported Configuration

Is it supported to configure both streams on the same Windows Event Log data source?

```json
[
"Microsoft-Event",
"Microsoft-WindowsEvent"
]
```

### Expected Behavior

If both streams are configured:
1. What is the expected ingestion behavior?
2. Is AMA expected to emit data to both streams?
3. Is one stream preferred over the other?
4. Can the selected stream change after an AMA upgrade?

### Stream Selection

For VMs with:

- identical DCRs
- identical DCR associations
- identical AMA configuration
- identical AMA versions

what determines whether events land in:

- `Event`
- `WindowsEvent`?

### AMA Upgrade Impact

Was there a known change in AMA behavior between versions approximately 1.39.x and 1.42.x that would affect:

- Windows Event Log collection
- stream selection
- table selection
- ingestion routing?

### Internal Stream Mapping

Is there documentation describing the relationship between:

```text
Microsoft-Event
Microsoft-WindowsEvent

GENERIC_EVENT_BLOB
SECURITY_WEF_EVENT_BLOB

Event
WindowsEvent
```

and how they map to each other?

## Expected Clarification

We would like to understand whether:

- this configuration is supported and behaves as designed,
- the observed behavior indicates an AMA issue,
- or we should use only a single stream (`Microsoft-Event` or `Microsoft-WindowsEvent`) for Windows Event Log collection.

Any clarification regarding expected behavior when both streams are configured on the same `windows_event_log` data source would be greatly appreciated.

Thank you!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.