elastic / elastic/logstash

Grok oniguruma pattern not parsing field and type (i.e. <throughputtime:int>x results in "int" field in Elasticsearch")

Open
#3,557 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

This Oniguruma pattern: (?throughputtime:int[0-9]+) now results in the field "int" in elasticsearch, and not "throughputtime" as an int type in elasticsearch.

We noticed this issue after upgrading from 1.4.2 to 1.5.2. The problem presented itself in 1.5.0 of logstash.

Replication is as follows:

message:

```
INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760
```

Config:

```

input {
stdin { }
}

filter {
grok {
patterns_dir => "/opt/logstash/patterns/"
match => [ "message", ".*pii\s+(?[^,]+),.*correlationId\s+(?[^,]+),.*throughput time\s+(?[0-9]+).*$"]
}
}

output {
stdout { codec => rubydebug }
}

```

Running in 1.4.2 results in the following:

```
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:32:49.411Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"throughputtime" => 1760
}
```

Running in 1.5.0 results in the following:

```
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:26:30.398Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"int" => "1760"
}
```

---

Simple command line examples:

1.4.3

```
echo "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760" | logstash-1.4.3/bin/logstash -f filter.conf
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:32:49.411Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"throughputtime" => 1760
}

```

1.5.0

```
echo "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760" | logstash-1.5.0/bin/logstash -f filter.conf
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:33:01.506Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"int" => "1760"
}
```

1.5.2

```
echo "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760" | logstash-1.5.2/bin/logstash -f filter.conf
{
"message" => "INFO [2015-07-03 01:46:13,420] Processor: Successfully dispatched message with pii 0022311562901666, correlationId 426f54b9-0dac-4c44-9d30-8cb815febe3f, and throughput time 1760",
"@version" => "1",
"@timestamp" => "2015-07-03T14:52:24.951Z",
"pii" => "0022311562901666",
"correlationId" => "426f54b9-0dac-4c44-9d30-8cb815febe3f",
"int" => "1760"
}
```

I've looked at the changelog.md and nothing indicate a breaking change with regards to GROK and this field matching and type specification. Is this a known issue?

thanks
/dom

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.