influxdata / influxdata/influxdb-plugin-fluent

Using a JSON key for the parameters tag_keys and field_keys

Open
#47 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
21
Forks
0
PR merge metrics
No merged PRs in 30d

Description

__Proposal:__
Is it possible to enhance the InfluxDB output plugin (https://github.com/influxdata/influxdb-plugin-fluent) to allow the tag_keys and field_keys settings to accept arrays specified within the JSON structure?

__Current behavior:__
Currently, only static configuration is possible.

__Desired behavior:__
The current static enumeration significantly limits flexibility, especially when dealing with data sources where tags and fields vary dynamically. While the measurement can be set dynamically, its utility is constrained since different measurements typically have distinct fields and tags.

For instance, consider a scenario where a JSON key contains an array of tags, and this key is referenced in the configuration:

{
"name": "apiserver_request_sli_duration_seconds_bucket",
"timestamp": "2025-02-17T14:49:04Z",
"value": "88808",
"labels_keys": [
"__name__",
"apiserver",
"component",
"endpoint",
"group",
"instance",
"job",
"le",
"namespace",
"prometheus",
"prometheus_replica",
"resource",
"scope",
"service",
"verb",
"version",
"topic"
],
"__name__": "apiserver_request_sli_duration_seconds_bucket",
"apiserver": "kube-apiserver",
"component": "apiserver",
"endpoint": "https",
"group": "machineconfiguration.openshift.io",
"instance": "1.1.1.12:6443",
"job": "apiserver",
"le": "3",
"namespace": "default",
"prometheus": "openshift-monitoring/k8s",
"prometheus_replica": "prometheus-k8s-0",
"resource": "machineconfigs",
"scope": "resource",
"service": "kubernetes",
"verb": "GET",
"version": "v1",
"topic": "ch-ocp-cw01-prod-ocp-metrics"
}

In this example, the labels_keys array enumerates the desired tags. Enhancing the plugin to interpret such arrays within the JSON payload would provide greater flexibility in handling dynamic data sources.

This feature request aligns with discussions in the plugin's GitHub repository, where users have expressed the need for dynamically setting tag_keys based on incoming data. Implementing this capability would significantly enhance the plugin's versatility in diverse logging environments.

__Alternatives considered:__
Describe other solutions or features you considered.

__Use case:__
Why is this important (helps with prioritizing requests)?

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the InfluxDB output plugin's configuration handling for tag_keys and field_keys, then trace how incoming JSON payloads are mapped into tags and fields. Use the existing static configuration behavior as the baseline; done means arrays named by JSON keys can define tags and fields while static configuration continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
json, ruby
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.