Grok oniguruma pattern not parsing field and type (i.e. <throughputtime:int>x results in "int" field in Elasticsearch")
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
This Oniguruma pattern: (?throughputtime:int[0-9]+) now results in the field "int" in elasticsearch, and not "throughputtime" as an int type in elasticsearch.
We noticed this issue after upgrading from 1.4.2 to 1.5.2. The problem presented itself in 1.5.0 of logstash.
Replication is as follows:
message:
```
INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760
```
Config:
```
input {
stdin { }
}
filter {
grok {
patterns_dir => "/opt/logstash/patterns/"
match => [ "message", ".*pii\s+(?[^,]+),.*correlationId\s+(?[^,]+),.*throughput time\s+(?[0-9]+).*$"]
}
}
output {
stdout { codec => rubydebug }
}
```
Running in 1.4.2 results in the following:
```
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:32:49.411Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"throughputtime" => 1760
}
```
Running in 1.5.0 results in the following:
```
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:26:30.398Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"int" => "1760"
}
```
---
Simple command line examples:
1.4.3
```
echo "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760" | logstash-1.4.3/bin/logstash -f filter.conf
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:32:49.411Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"throughputtime" => 1760
}
```
1.5.0
```
echo "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760" | logstash-1.5.0/bin/logstash -f filter.conf
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:33:01.506Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"int" => "1760"
}
```
1.5.2
```
echo "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760" | logstash-1.5.2/bin/logstash -f filter.conf
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:52:24.951Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"int" => "1760"
}
```
I've looked at the changelog.md and nothing indicate a breaking change with regards to GROK and this field matching and type specification. Is this a known issue?
thanks
/dom
Contributor guide
Assessment
This issue has not been assessed yet.