elastic / elastic/beats

[Bug] [auditbeat] [winlogbeat] Running `setup` with `setup.ilm.check_exists: false` creates invalid mappings (`text` for `keyword` fields), effecting the Security app

Open
#21,725 14 comments 2 reactions 1 assignee Claimed by @andrewkroh View on GitHub
Auditbeat bug libbeat Team:Security-Service Integrations Winlogbeat
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

### Running `setup` with `setup.ilm.check_exists: false` creates invalid mappings (`text` for `keyword` fields), effecting the Security app

- Version: `7.9`, `7.10 BC1`
- Operating System: `Windows 10`, `macOS Catalina 10.15.7`
- Discuss Forum URL: https://discuss.elastic.co/t/elastic-7-9-1-security-siem-your-visualization-has-error-s-illegal-argument-exception/250265

As [reported by a user in Discuss](https://discuss.elastic.co/t/elastic-7-9-1-security-siem-your-visualization-has-error-s-illegal-argument-exception/250265) against `7.9.1` via `winlogbeat`, and reproducable in `7.10 BC1` with both `winlogbeat` and `auditbeat`, running:

```
winlogbeat setup
```

(as reported in `7.9.1`, and reproducible in `7.10 BC1`)

or

```
auditbeat setup
```

in `7.10 BC1` with the following configuration in `winlogbeat.yml` or `auditbeat.yml`:

```
setup.ilm.check_exists: false
```

creates incorrect mappings with `type` `text` for some `keyword` fields. These incorrect mappings effect the Security app in the following ways:

### Errors in `7.9`

- As [reported in Discuss](https://discuss.elastic.co/t/elastic-7-9-1-security-siem-your-visualization-has-error-s-illegal-argument-exception/250265) against `7.9.1`:

> We have a brand new cluster install to test 7.9.1 and the security features in particular.
>
> When navigating to the Overview page we get the following error.
>
```
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.module] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
```

![text-fields-are-not-optimised-for-operations](https://user-images.githubusercontent.com/4459398/95784934-06933000-0c92-11eb-8f78-e7783aea7ebc.png)

### No hosts in the `Hosts` page in `7.10 BC1`

Per the screenshot below, in `7.10 BC1`, the `Hosts` page will not list any hosts, even though searching for `host.name : *` in a Timeline returns results for the same period (Last 24 hours):

no-hosts

### Why users may add this setting

In `7.10 BC1` when, for example, `auditbeat setup` is run, the following output is displayed:

```
➜ auditbeat-7.10.0-darwin-x86_64 sudo ./auditbeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
```

Some users may see

```
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
```

in the output above and explicitly negate the `setup.ilm.check_exist` setting by adding

```
setup.ilm.check_exists: false
```

to their config.

### Mapping Errors

When `setup` is run _without_ including

```
setup.ilm.check_exists: false
```

in the configuration, the following query run from `Dev Tools`:

```
GET /auditbeat-7.10.0/_mapping
```

returns the expected mapping for `event.module`:

```
...
"event" : {
"properties" : {
"action" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
"module" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
}
}
},
```

When `auditbeat setup` is run with `setup.ilm.check_exists: false` in `auditbeat.yml` in `7.10 BC1`, the mapping for `event.module` returned by `GET /auditbeat-7.10.0/_mapping` is:

```
...
"event" : {
"properties" : {
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
}
},
```

Note the extra (incorrect):

```
"type" : "text",
```

in the definition of `event.module`.

### Reproduction steps

1) Download the latest version of auditbeat for `7.10 BC1`

2) Spin up a new `7.10 BC1` instance on Elastic Cloud

3) Login to Kibana, and navigate to the Security app

4) Click the `Add data` button

5) Download the version of `auditbeat` built for `7.10 BC1`

6) After extracting the archive, edit `auditbeat.yml`

7) In addition to your Elastic Cloud credentials, add the following setting to `auditbeat.yml`:

```
setup.ilm.check_exists: false
```

8) Run

```
auditbeat setup
```

9) Ingest data by running

```
auditbeat -e
```

10) Click the `Check data` button on the `Auditbeat` add data page to confirm that Data was successfully received.

11) From the main Kibana menu, Navigate to the `Security > Hosts` page

**Expected result**

* The host you ingested `Auditbeat` data from is displayed

**Actual result**

* Per the screenshot below, the `Hosts` page shows `0` hosts:

no-hosts

12) Click the `Timeline` flyout to open a Timeline

13) Enter the following KQL:

```
host.name : *
```

**Expected result**

* For the same date range (`Last 24 hours`), data from the host where you ran `auditbeat` appears.

14) In Kibana `Dev Tools`, execute the following command:

```
GET /auditbeat-7.10.0/_mapping
```

**Expected result**

- `GET /auditbeat-7.10.0/_mapping` returns the expected mapping for `event.module`:

```
...
"event" : {
"properties" : {
"action" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
"module" : {
"type" : "keyword",
"ignore_above" : 1024
},
...
}
}
},
```

**Actual result**

- The mapping returned by `GET /auditbeat-7.10.0/_mapping`:

```
...
"event" : {
"properties" : {
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
}
},
```

contains an extra (incorrect):

```
"type" : "text",
```

in the definition of `event.module`.

CC @polyfractal

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.