elastic / elastic/integrations

[Azure Logs] Mapping conflict an dropped events because of field that can be both a string and an object.

Open
#14,178 4 comments 10 reactions 0 assignees View on GitHub
Integration:azure_logs
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Hello,

We just found out that some events for the Azure Signin Entra ID logs are being dropped because one of the fields can be sent by Microsoft as both a string or list and also as an object.

The resulting offending field is the field `azure.signinlogs.properties.conditional_access_audiences`, that is parsed from the field `conditionalAccessAudiences` in the siginlogs dataset.

It seems that Microsoft can send events where this field has conflicting types, on some events it looks like this:

```
{
"conditionalAccessAudiences": ["list-with-1-or-more-items"]
}
```

And in other events it looks like this

```
{
"conditionalAccessAudiences": {
"applicationId": "some-value",
"audienceReasons": "some-values"
}
}
```

It also seems that Microsoft made some changes on their side and stopped sending the information for the `audienceReasons` and also started to send the value of the `applicationId` directly on the `conditionalAccessAudiences` field, which leads to the conflict.

This field also does not have explict mapping on Elasticsearch side, so it will be dynamically mapped according to the first event it gets, so this lead to a conflict in Kibana as well in my case as one of the data streams got the mapping for this field as `keyword` instead of the object.

I was able to stop the events from being dropped by explictly mapping the field and using a custom ingest pipeline to rename the cases where `azure.signinlogs.properties.conditional_access_audiences` is a list into `azure.signinlogs.properties.conditional_access_audiences.application_id`.

This is the custom mapping I'm using:

```
PUT _component_template/logs-azure.signinlogs@custom
{
"template": {
"mappings": {
"properties": {
"azure": {
"type": "object",
"properties": {
"signinlogs": {
"type": "object",
"properties": {
"properties": {
"type": "object",
"properties": {
"conditional_access_audiences": {
"type": "object",
"properties": {
"application_id": {
"type": "keyword"
},
"audience_reasons": {
"type": "keyword"
}
}
}
}
}
}
}
}
}
}
}
}
}
```

And this is the custom ingest pipeline:

```
PUT _ingest/pipeline/logs-azure.signinlogs@custom
{
"description": "custom signinlogs",
"processors": [
{
"rename": {
"ignore_failure": true,
"ignore_missing": true,
"field": "azure.signinlogs.properties.conditional_access_audiences",
"target_field": "azure.signinlogs.properties.conditional_access_audiences.application_id",
"if": "ctx?.azure?.signinlogs?.properties?.conditional_access_audiences instanceof List"
}
}
]
}
```

This should be applied to the Ingest pipeline and mapping for the integration.

Another issue is that this conflict may break a bult-in security rule: [Entra ID Device Code Auth with Broker Client](https://github.com/elastic/detection-rules/blob/main/rules/integrations/azure/credential_access_entra_id_device_code_auth_with_broker_client.toml)

This rule uses the field `azure.signinlogs.properties.conditional_access_audiences.application_id`, and since this can lead to a mapping conflict in Kibana the queries using this field may not work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.