elastic / elastic/integrations

[System]: Multiple spaces in GROK pattern

Open
#15,818 2 comments 0 reactions 0 assignees View on GitHub
Integration:system Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

System [system]

### Dataset Name

system.auth

### Integration Version

2.6.1

### Agent Version

9.2

### Agent Output Type

elasticsearch

### Elasticsearch Version

9.2

### OS Version and Architecture

Debian 12

### Software/API Version

_No response_

### Error Message

There is no error message

### Event Original

_No response_

### What did you do?

We're working on a script that checks if there have been modifications within default managed ES pipelines.

### What did you see?

While working on the script, it reported a difference in the `logs-system.auth-{version}-log` pipeline. The difference being in the first (grok) processor and it's first pattern starting on [line 13](https://github.com/elastic/integrations/blob/main/packages/system/data_stream/auth/elasticsearch/ingest_pipeline/log.yml#L13):
```
{
"name": "logs-system.auth-2.6.1-log",
"status": "different",
"file": "./epr_packages/system-2.6.1/system-2.6.1/data_stream/auth/elasticsearch/ingest_pipeline/log.yml",
"diff": {
"values_changed": {
"root['processors'][0]['grok']['patterns'][0]": {
"new_value": "^<%{NONNEGINT:log.syslog.priority:int}>(?:%{NONNEGINT:system.auth.syslog.version} )?+(?:-|%{TIMESTAMP:system.auth.timestamp}) +(?:-|%{IPORHOST:host.hostname}) +(?:-|%{SYSLOG5424PRINTASCII:process.name}) +(?:-|%{POSINT:process.pid:long}) +(?:-|%{SYSLOG5424PRINTASCII:event.code}) +(?:-|%{SYSLOG5424SD:syslog5424_sd})? +%{GREEDYDATA:_temp.message}$",
"old_value": "^<%{NONNEGINT:log.syslog.priority:int}>(?:%{NONNEGINT:system.auth.syslog.version} )?+(?:-|%{TIMESTAMP:system.auth.timestamp}) +(?:-|%{IPORHOST:host.hostname}) +(?:-|%{SYSLOG5424PRINTASCII:process.name}) +(?:-|%{POSINT:process.pid:long}) +(?:-|%{SYSLOG5424PRINTASCII:event.code}) +(?:-|%{SYSLOG5424SD:syslog5424_sd})? +%{GREEDYDATA:_temp.message}$"
}
}
}
}
```

We are reading the python with pyyaml and it doesn't read the literal spaces at the end and at the front of the yaml:
```
$ curl -s https://raw.githubusercontent.com/elastic/integrations/refs/heads/main/packages/system/data_stream/auth/elasticsearch/ingest_pipeline/log.yml | head -20 | cat -A
---$
description: Pipeline for parsing system authorization and secure logs.$
processors:$
- grok:$
if: ctx.log?.syslog == null$
description: Grok the message header.$
tag: grok-message-header$
field: event.original$
pattern_definitions:$
GREEDYMULTILINE: '(.|\n)*'$
TIMESTAMP: (?:%{TIMESTAMP_ISO8601}|%{SYSLOGTIMESTAMP})$
patterns:$
- '^<%{NONNEGINT:log.syslog.priority:int}>(?:%{NONNEGINT:system.auth.syslog.version} )?+(?:-|%{TIMESTAMP:system.auth.timestamp}) $
+(?:-|%{IPORHOST:host.hostname}) +(?:-|%{SYSLOG5424PRINTASCII:process.name}) $
+(?:-|%{POSINT:process.pid:long}) +(?:-|%{SYSLOG5424PRINTASCII:event.code}) $
+(?:-|%{SYSLOG5424SD:syslog5424_sd})? $
+%{GREEDYDATA:_temp.message}$'$
- '^%{TIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname}? %{DATA:process.name}(?:\[%{POSINT:process.pid:long}\])?:%{SPACE}%{GREEDYMULTILINE:_temp.message}$'$
...
```

### What did you expect to see?

I'm not sure. We don't know if this is by design, or an issue with the yaml (trailing spaces), or an issue with your yaml parser/converted in the process of creating the ingest pipeline.
If this is by design, feel free to close this issue, we'll update our script to work accordingly.

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.