[Bug] [auditbeat] [winlogbeat] Running `setup` with `setup.ilm.check_exists: false` creates invalid mappings (`text` for `keyword` fields), effecting the Security app
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
### Running `setup` with `setup.ilm.check_exists: false` creates invalid mappings (`text` for `keyword` fields), effecting the Security app
- Version: `7.9`, `7.10 BC1`
- Operating System: `Windows 10`, `macOS Catalina 10.15.7`
- Discuss Forum URL: https://discuss.elastic.co/t/elastic-7-9-1-security-siem-your-visualization-has-error-s-illegal-argument-exception/250265
As [reported by a user in Discuss](https://discuss.elastic.co/t/elastic-7-9-1-security-siem-your-visualization-has-error-s-illegal-argument-exception/250265) against `7.9.1` via `winlogbeat`, and reproducable in `7.10 BC1` with both `winlogbeat` and `auditbeat`, running:
```
winlogbeat setup
```
(as reported in `7.9.1`, and reproducible in `7.10 BC1`)
or
```
auditbeat setup
```
in `7.10 BC1` with the following configuration in `winlogbeat.yml` or `auditbeat.yml`:
```
setup.ilm.check_exists: false
```
creates incorrect mappings with `type` `text` for some `keyword` fields. These incorrect mappings effect the Security app in the following ways:
### Errors in `7.9`
- As [reported in Discuss](https://discuss.elastic.co/t/elastic-7-9-1-security-siem-your-visualization-has-error-s-illegal-argument-exception/250265) against `7.9.1`:
> We have a brand new cluster install to test 7.9.1 and the security features in particular.
>
> When navigating to the Overview page we get the following error.
>
```
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.module] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
```

### No hosts in the `Hosts` page in `7.10 BC1`
Per the screenshot below, in `7.10 BC1`, the `Hosts` page will not list any hosts, even though searching for `host.name : *` in a Timeline returns results for the same period (Last 24 hours):

### Why users may add this setting
In `7.10 BC1` when, for example, `auditbeat setup` is run, the following output is displayed:
```
➜ auditbeat-7.10.0-darwin-x86_64 sudo ./auditbeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
```
Some users may see
```
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
```
in the output above and explicitly negate the `setup.ilm.check_exist` setting by adding
```
setup.ilm.check_exists: false
```
to their config.
### Mapping Errors
When `setup` is run _without_ including
```
setup.ilm.check_exists: false
```
in the configuration, the following query run from `Dev Tools`:
```
GET /auditbeat-7.10.0/_mapping
```
returns the expected mapping for `event.module`:
```
...
"event" : {
"properties" : {
"action" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
"module" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
}
}
},
```
When `auditbeat setup` is run with `setup.ilm.check_exists: false` in `auditbeat.yml` in `7.10 BC1`, the mapping for `event.module` returned by `GET /auditbeat-7.10.0/_mapping` is:
```
...
"event" : {
"properties" : {
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
}
},
```
Note the extra (incorrect):
```
"type" : "text",
```
in the definition of `event.module`.
### Reproduction steps
1) Download the latest version of auditbeat for `7.10 BC1`
2) Spin up a new `7.10 BC1` instance on Elastic Cloud
3) Login to Kibana, and navigate to the Security app
4) Click the `Add data` button
5) Download the version of `auditbeat` built for `7.10 BC1`
6) After extracting the archive, edit `auditbeat.yml`
7) In addition to your Elastic Cloud credentials, add the following setting to `auditbeat.yml`:
```
setup.ilm.check_exists: false
```
8) Run
```
auditbeat setup
```
9) Ingest data by running
```
auditbeat -e
```
10) Click the `Check data` button on the `Auditbeat` add data page to confirm that Data was successfully received.
11) From the main Kibana menu, Navigate to the `Security > Hosts` page
**Expected result**
* The host you ingested `Auditbeat` data from is displayed
**Actual result**
* Per the screenshot below, the `Hosts` page shows `0` hosts:

12) Click the `Timeline` flyout to open a Timeline
13) Enter the following KQL:
```
host.name : *
```
**Expected result**
* For the same date range (`Last 24 hours`), data from the host where you ran `auditbeat` appears.
14) In Kibana `Dev Tools`, execute the following command:
```
GET /auditbeat-7.10.0/_mapping
```
**Expected result**
- `GET /auditbeat-7.10.0/_mapping` returns the expected mapping for `event.module`:
```
...
"event" : {
"properties" : {
"action" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
"module" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
}
}
},
```
**Actual result**
- The mapping returned by `GET /auditbeat-7.10.0/_mapping`:
```
...
"event" : {
"properties" : {
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
}
},
```
contains an extra (incorrect):
```
"type" : "text",
```
in the definition of `event.module`.
CC @polyfractal
Contributor guide
Assessment
This issue has not been assessed yet.