influxdata / influxdata/influxdb
Data loss in influxdb due to aggregators
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
I am pushing data from NATS to InfluxDB using Telegraf. The data is being generated continuously, which results in multiple entries having the same timestamp. My timestamps are in nanoseconds, but in InfluxDB, the data is being aggregated, causing a loss of data points in my Grafana dashboard.
Can any one suggest how to avoid data loss.
example data.
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test time=1728981942717004500,count=1,errorCount=0,sentBytes=1467,timeStamp=1728981940732,receivedBytes=59275,latency=0,processingTime=0,connectTime=0,responseTime=3151 **1728981942717004544**
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test latency=0,receivedBytes=59275,time=1728981942717004500,errorCount=0,responseTime=1276,count=1,processingTime=0,sentBytes=1467,timeStamp=1728981942619,connectTime=0 **1728981942717004544**
Below is my telegraf configuration
[[inputs.nats_consumer]]
servers = [“nats://localhost:4222”]
subjects = [“nats.subject.requestsraw”]
data_format = “xpath_json”
xpath_native_types = true
fieldexclude = [“subjects”, “runId”, “testName”, “nodeName”, “ObjectType”, “errorMessage”, “errorResponseBody”, “requestName”, “responseCode”, “result”, “samplerType”]
[[inputs.nats_consumer.xpath]]
metric_name = “‘requestsRaw’”
metric_selection = “/"
timestamp = “time”
timestamp_format = “unix_ns”
field_selection = "”
[inputs.nats_consumer.xpath.tags]
runId = “string(/runId)”
testName = “string(/testName)”
nodeName = “string(/nodeName)”
ObjectType = “string(/ObjectType)”
errorMessage = “string(/errorMessage)”
errorResponseBody = “string(/errorResponseBody)”
requestName = “string(/requestName)”
responseCode = “string(/responseCode)”
result = “string(/result)”
samplerType = “string(/samplerType)”
Contributor guide
Research direction
Start with the Telegraf [[inputs.nats_consumer]] and [[inputs.nats_consumer.xpath]] configuration, especially the timestamp and field selections. Reproduce the two sample points with the same timestamp and inspect how InfluxDB identifies series and handles duplicate timestamps. Done means establishing whether the behavior is expected and documenting a supported way to preserve both points.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana
- Domain
- databases, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100