influxdata / influxdata/influxdb
How to to parse the telegraf data
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
we are pushing the data from NATS server to influxDb using telegraf. The telegraf is parsing the NATS (json array) to required format. But still we are not receiving the data to influx. if any can tell me how i can parse the data.
Below is my telegraf configuration, NATS output json array and telegraf output.
[[inputs.nats_consumer]]
servers = ["nats://localhost:4222"]
subjects = ["nats.subject.requestsraw"]
data_format = "json_v2"
name_override = "requestsRaw"
# Time-related settings
json_time_key = "time"
json_time_format = "unix_ns" # Unix timestamp in nanoseconds
tagexclude = ["_time"]
[[inputs.nats_consumer.json_v2]]
[[inputs.nats_consumer.json_v2.object]]
path = "@this"
disable_prepend_keys = true
tags = ["runId", "testName", "nodeName", "ObjectType", "errorMessage", "errorResponseBody", "requestName", "responseCode", "result", "samplerType"]
# Field mappings (convert fields to correct types)
[inputs.nats_consumer.json_v2.object.fields]
connectTime = "integer"
count = "integer"
errorCount = "integer"
latency = "integer"
processingTime = "integer"
receivedBytes = "integer"
sentBytes = "integer"
responseTime = "integer"
nats json arry
{"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1865,"timeStamp":1728279893708,"time":1728279894329000300,"ObjectType":"requestsRaw"}, {"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1552,"timeStamp":1728279894021,"time":1728279894329000400,"ObjectType":"requestsRaw"}, {"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1448,"timeStamp":1728279894125,"time":1728279894329000500,"ObjectType":"requestsRaw"}, {"runId":"R_20241007-1114","testName":"Test","nodeName":"Test-Node","errorMessage":"noData","errorResponseBody":"noData","requestName":"WebSocket request-response Sampler","responseCode":"200","result":"pass","samplerType":"transaction","connectTime":0,"count":1,"errorCount":0,"latency":0,"processingTime":0,"receivedBytes":59275,"sentBytes":1467,"responseTime":1425,"timeStamp":1728279894148,"time":1728279894329000600,"ObjectType":"requestsRaw"}]
telegraf output
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1446,timeStamp=1728279894126,time=1728279894329000200 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1913,timeStamp=1728279893660,time=1728279894329000200 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1865,timeStamp=1728279893708,time=1728279894329000200 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1552,timeStamp=1728279894021,time=1728279894329000400 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1448,timeStamp=1728279894125,time=1728279894329000400 1728279895627287728
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241007-1114,samplerType=transaction,testName=Test connectTime=0,count=1,errorCount=0,latency=0,processingTime=0,receivedBytes=59275,sentBytes=1467,responseTime=1425,timeStamp=1728279894148,time=1728279894329000700 1728279895627287728
Contributor guide
Research direction
Start by reproducing the supplied inputs.nats_consumer configuration and comparing the NATS JSON array with the shown Telegraf line protocol output. Check whether the json_v2 parsing and time-related settings account for the missing InfluxDB data. Done means identifying the ingestion failure and confirming that the resulting points are accepted by InfluxDB.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100