influxdata / influxdata/influxdb

How to to parse the telegraf data

Open
#25,430 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

we are pushing the data from NATS server to influxDb using telegraf. The telegraf is parsing the NATS (json array) to required format. But still we are not receiving the data to influx. if any can tell me how i can parse the data.

Below is my telegraf configuration, NATS output json array and telegraf output.

[[inputs.nats_consumer]]
servers = ["nats://localhost:4222"]
subjects = ["nats.subject.requestsraw"]
data_format = "json_v2"
name_override = "requestsRaw"

# Time-related settings
json_time_key = "time"
json_time_format = "unix_ns" # Unix timestamp in nanoseconds
tagexclude = ["_time"]

[[inputs.nats_consumer.json_v2]]
[[inputs.nats_consumer.json_v2.object]]
path = "@this"
disable_prepend_keys = true
tags = ["runId", "testName", "nodeName", "ObjectType", "errorMessage", "errorResponseBody", "requestName", "responseCode", "result", "samplerType"]

# Field mappings (convert fields to correct types)
[inputs.nats_consumer.json_v2.object.fields]
connectTime = "integer"
count = "integer"
errorCount = "integer"
latency = "integer"
processingTime = "integer"
receivedBytes = "integer"
sentBytes = "integer"
responseTime = "integer"

nats json arry

{"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1865,"timeStamp":1728279893708,"time":1728279894329000300,"ObjectType":"requestsRaw"}, {"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1552,"timeStamp":1728279894021,"time":1728279894329000400,"ObjectType":"requestsRaw"}, {"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1448,"timeStamp":1728279894125,"time":1728279894329000500,"ObjectType":"requestsRaw"}, {"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1425,"timeStamp":1728279894148,"time":1728279894329000600,"ObjectType":"requestsRaw"}]

telegraf output

requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1446,timeStamp=1728279894126,time=1728279894329000200 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1913,timeStamp=1728279893660,time=1728279894329000200 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1865,timeStamp=1728279893708,time=1728279894329000200 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1552,timeStamp=1728279894021,time=1728279894329000400 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1448,timeStamp=1728279894125,time=1728279894329000400 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1425,timeStamp=1728279894148,time=1728279894329000700 1728279895627287728

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied inputs.nats_consumer configuration and comparing the NATS JSON array with the shown Telegraf line protocol output. Check whether the json_v2 parsing and time-related settings account for the missing InfluxDB data. Done means identifying the ingestion failure and confirming that the resulting points are accepted by InfluxDB.

Written by the indexing model from the issue text.

Assessment

Domain
databases, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.