influxdata / influxdata/influxdb

Data loss in influxdb due to aggregators

Open
#25,469 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I am pushing data from NATS to InfluxDB using Telegraf. The data is being generated continuously, which results in multiple entries having the same timestamp. My timestamps are in nanoseconds, but in InfluxDB, the data is being aggregated, causing a loss of data points in my Grafana dashboard.

Can any one suggest how to avoid data loss.

example data.

requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test time=1728981942717004500,count=1,errorCount=0,sentBytes=1467,timeStamp=1728981940732,receivedBytes=59275,latency=0,processingTime=0,connectTime=0,responseTime=3151 **1728981942717004544**
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test latency=0,receivedBytes=59275,time=1728981942717004500,errorCount=0,responseTime=1276,count=1,processingTime=0,sentBytes=1467,timeStamp=1728981942619,connectTime=0 **1728981942717004544**

Below is my telegraf configuration

[[inputs.nats_consumer]]
servers = [“nats://localhost:4222”]
subjects = [“nats.subject.requestsraw”]
data_format = “xpath_json”
xpath_native_types = true
fieldexclude = [“subjects”, “runId”, “testName”, “nodeName”, “ObjectType”, “errorMessage”, “errorResponseBody”, “requestName”, “responseCode”, “result”, “samplerType”]

[[inputs.nats_consumer.xpath]]
metric_name = “‘requestsRaw’”
metric_selection = “/"
timestamp = “time”
timestamp_format = “unix_ns”
field_selection = "”
[inputs.nats_consumer.xpath.tags]
runId = “string(/runId)”
testName = “string(/testName)”
nodeName = “string(/nodeName)”
ObjectType = “string(/ObjectType)”
errorMessage = “string(/errorMessage)”
errorResponseBody = “string(/errorResponseBody)”
requestName = “string(/requestName)”
responseCode = “string(/responseCode)”
result = “string(/result)”
samplerType = “string(/samplerType)”

Contributor guide

Open the contributing guide

Research direction

Start with the Telegraf [[inputs.nats_consumer]] and [[inputs.nats_consumer.xpath]] configuration, especially the timestamp and field selections. Reproduce the two sample points with the same timestamp and inspect how InfluxDB identifies series and handles duplicate timestamps. Done means establishing whether the behavior is expected and documenting a supported way to preserve both points.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.