influxdata / influxdata/telegraf

[parser.enum] regex support

Open
#18,284 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

Use Case

Using telegraf to parse logs using [inputs.tail] and grok and then having to normalise data, especially constants, I am finding I need to write long lists of regex using [processor.regex]. It would be real nice if I could just use [processor.enum] to do all the normalisation using a regex - nice and clean. This would have to be an optional flag on the enum processor so as not to break backwards compatibility.

Expected behavior

Something like

  [processors.enum.mapping.value_mappings]
    "^72-20." = "ConfigAgent"
.
.
.
Actual behavior

Having regex on enum would replace long lists of this and make it much more readable:

    # License and configuration events (72-20x, 21-*, 05-*, 51-*, 04-*)
    [[processors.regex.fields]]
      key = "process"
      pattern = '^72-20.*'
      replacement = "ConfigAgent"

    # Port and interface events (62-*, 72-*, 20-*)
    [[processors.regex.fields]]
      key = "process"
      pattern = '^62-.*|^72-.*|^20-.*'
      replacement = "Ebra"

    # Authentication and authorization events (46-*, 24-*)
    [[processors.regex.fields]]
      key = "process"
      pattern = '^46-.*|^24-.*'
      replacement = "Aaa"

    # Configuration management events (21-*, 05-*, 51-*, 04-*)
    [[processors.regex.fields]]
      key = "process"
      pattern = '^21-.*|^05-.*|^51-.*|^04-.*'
      replacement = "ConfigAgent"

    # Performance monitoring events (80-*)
    [[processors.regex.fields]]
      key = "process"
      pattern = '^80-.*'
      replacement = "Tca"
.
.
.
Additional info

This will only work for me if I can do inplace field replacements

  [[processors.enum]]
    [[processors.enum.mapping]]
      field = "status"
      dest = "status"  # same as source

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the processors.enum and processors.regex implementations and their configuration handling. Compare enum mapping behavior with regex replacement, including the requested same-source destination, and define how an optional regex mode preserves existing mappings. Done means regex mappings can normalize fields in place without changing default enum behavior, with coverage for the examples described.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.