influxdata / influxdata/telegraf

Add new parser to be abble to simply count lines that match pattern

Open
#16,033 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Be abble to simply count number of line in a stream that match or not a pattern.
I will developpe this plugin, but first I share the goal.

Sample of specification :

# Pattern Parser Plugin

The `pattern` parser creates metrics from a stream containing lines.
It counts number of lines matching a pattern.

## Configuration

```toml
[[inputs.file]]
files = ["/tmp/test.log"]

## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "pattern"

## This is a list of searches to check the given stream for.
## An entry can have the following properties:
## tag_name -- Name of the tag added to the metric.
## tag_value -- Value of the tag added to the metric.
## pattern -- Pattern searched in the stream.
pattern_searches = [
{ tag_name = "severite", tag_value = "error", pattern = "error" },
{ tag_name = "severite", tag_value = "warning", pattern = "[Ww]arning" },
{ tag_name = "severite", tag_value = "connection timeout", pattern = "connection to .* timeout" },
]
```

## Metrics

One metric is created for each search with tag "tag_name" contain "tag_value".

## Examples

Config:

```toml
[[inputs.file]]
files = ["example"]
data_format = "pattern"
pattern_searches = [
{ tag_name = "status", tag_value = "Job success", pattern = "Job successfully completed" }
]
```

Input:

```text
Job failed to run
```

Output:

```text
file_pattern,status=Job\ success match_count=0,not_match_count=1
```

Config:

```toml
[[inputs.file]]
files = ["example"]
data_format = "pattern"
pattern_searches = [
{ tag_name = "error_code", tag_value = "2XX", pattern = "^([^ ]* ){3}\[.*\] "[^"]*" (2[0-9][0-9]) .*$" },
{ tag_name = "error_code", tag_value = "3XX", pattern = "^([^ ]* ){3}\[.*\] "[^"]*" (3[0-9][0-9]) .*$" },
{ tag_name = "error_code", tag_value = "4XX", pattern = "^([^ ]* ){3}\[.*\] "[^"]*" (4[0-9][0-9]) .*$" },
{ tag_name = "error_code", tag_value = "5XX", pattern = "^([^ ]* ){3}\[.*\] "[^"]*" (5[0-9][0-9]) .*$" },
]
```

Input:

```text
10.158.236.103 - - [15/Oct/2024:13:58:15 +0200] "GET / HTTP/1.0" 200 14 2195 0 - "-" "-" -
10.158.236.103 - - [15/Oct/2024:13:58:15 +0200] "POST / HTTP/1.0" 201 14 2195 0 - "-" "-" -
10.158.236.103 - - [15/Oct/2024:13:58:15 +0200] "GET /test.html HTTP/1.0" 500 14 2195 0 - "-" "-" -
10.158.236.103 - - [15/Oct/2024:13:58:15 +0200] "GET /login HTTP/1.0" 400 14 2195 0 - "-" "-" -
10.158.236.103 - - [15/Oct/2024:13:59:15 +0200] "GET / HTTP/1.0" 200 14 2195 0 - "-" "-" -
```

Output:

```text
file_pattern,error_code=2XX match_count=3,not_match_count=2
file_pattern,error_code=3XX match_count=0,not_match_count=5
file_pattern,error_code=4XX match_count=1,not_match_count=4
file_pattern,error_code=5XX match_count=1,not_match_count=4
```

### Expected behavior

Have a simple plugin to count lines that match a pattern.

### Actual behavior

In fact, some use cases are possible by combining grok and aggregator, but it is very heavy to implement.
It is also very difficult to configure these plugins well, especially with logs whose content is not precisely structured. For example, counting the words "Error" anywhere in a string.
Finally, if no line matches, no value is returned.

### Additional info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the data format guidance in DATA_FORMATS_INPUT.md and the inputs.file configuration examples in this issue. Compare the proposed pattern parser behavior with the existing grok and aggregator approaches mentioned here. Done means a defined parser configuration can report match_count and not_match_count for each search, including when no lines match.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.