elastic / elastic/integrations
[System]: Multiple spaces in GROK pattern
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
System [system]
### Dataset Name
system.auth
### Integration Version
2.6.1
### Agent Version
9.2
### Agent Output Type
elasticsearch
### Elasticsearch Version
9.2
### OS Version and Architecture
Debian 12
### Software/API Version
_No response_
### Error Message
There is no error message
### Event Original
_No response_
### What did you do?
We're working on a script that checks if there have been modifications within default managed ES pipelines.
### What did you see?
While working on the script, it reported a difference in the `logs-system.auth-{version}-log` pipeline. The difference being in the first (grok) processor and it's first pattern starting on [line 13](https://github.com/elastic/integrations/blob/main/packages/system/data_stream/auth/elasticsearch/ingest_pipeline/log.yml#L13):
```
{
"name": "logs-system.auth-2.6.1-log",
"status": "different",
"file": "./epr_packages/system-2.6.1/system-2.6.1/data_stream/auth/elasticsearch/ingest_pipeline/log.yml",
"diff": {
"values_changed": {
"root['processors'][0]['grok']['patterns'][0]": {
"new_value": "^<%{NONNEGINT:log.syslog.priority:int}>(?:%{NONNEGINT:system.auth.syslog.version} )?+(?:-|%{TIMESTAMP:system.auth.timestamp}) +(?:-|%{IPORHOST:host.hostname}) +(?:-|%{SYSLOG5424PRINTASCII:process.name}) +(?:-|%{POSINT:process.pid:long}) +(?:-|%{SYSLOG5424PRINTASCII:event.code}) +(?:-|%{SYSLOG5424SD:syslog5424_sd})? +%{GREEDYDATA:_temp.message}$",
"old_value": "^<%{NONNEGINT:log.syslog.priority:int}>(?:%{NONNEGINT:system.auth.syslog.version} )?+(?:-|%{TIMESTAMP:system.auth.timestamp}) +(?:-|%{IPORHOST:host.hostname}) +(?:-|%{SYSLOG5424PRINTASCII:process.name}) +(?:-|%{POSINT:process.pid:long}) +(?:-|%{SYSLOG5424PRINTASCII:event.code}) +(?:-|%{SYSLOG5424SD:syslog5424_sd})? +%{GREEDYDATA:_temp.message}$"
}
}
}
}
```
We are reading the python with pyyaml and it doesn't read the literal spaces at the end and at the front of the yaml:
```
$ curl -s https://raw.githubusercontent.com/elastic/integrations/refs/heads/main/packages/system/data_stream/auth/elasticsearch/ingest_pipeline/log.yml | head -20 | cat -A
---$
description: Pipeline for parsing system authorization and secure logs.$
processors:$
- grok:$
if: ctx.log?.syslog == null$
description: Grok the message header.$
tag: grok-message-header$
field: event.original$
pattern_definitions:$
GREEDYMULTILINE: '(.|\n)*'$
TIMESTAMP: (?:%{TIMESTAMP_ISO8601}|%{SYSLOGTIMESTAMP})$
patterns:$
- '^<%{NONNEGINT:log.syslog.priority:int}>(?:%{NONNEGINT:system.auth.syslog.version} )?+(?:-|%{TIMESTAMP:system.auth.timestamp}) $
+(?:-|%{IPORHOST:host.hostname}) +(?:-|%{SYSLOG5424PRINTASCII:process.name}) $
+(?:-|%{POSINT:process.pid:long}) +(?:-|%{SYSLOG5424PRINTASCII:event.code}) $
+(?:-|%{SYSLOG5424SD:syslog5424_sd})? $
+%{GREEDYDATA:_temp.message}$'$
- '^%{TIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname}? %{DATA:process.name}(?:\[%{POSINT:process.pid:long}\])?:%{SPACE}%{GREEDYMULTILINE:_temp.message}$'$
...
```
### What did you expect to see?
I'm not sure. We don't know if this is by design, or an issue with the yaml (trailing spaces), or an issue with your yaml parser/converted in the process of creating the ingest pipeline.
If this is by design, feel free to close this issue, we'll update our script to work accordingly.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.