influxdata / influxdata/influxdb

prometheus measurements not being recorded

Open
#20,336 0 comments 0 reactions 0 assignees View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. configure telegraf to scrape some prometheus endpoints (in this case, lighthouse beacon and validator)
2. attempt to graph them in chronograf

__Expected behavior:__
Data gets recorded, graphs show things

__Actual behavior:__
Data gets scraped by telegraf, gets submitted to influxdb (confirmed by tcpdump and http `write-tracing` in `influxdb.conf`), but only the first few data points seem to make it to disk; there is no error message, things just get dropped silently (this has been collecting data for 6 days, there should be way more points):

```
name: beacon_head_state_total_validators_total
time gauge host url
---- ----- ---- ---
2020-12-09T17:01:10Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:20Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:30Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:40Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:50Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:02:00Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:02:10Z 39885 eth2v1 http://127.0.0.1:5054/metrics
> select count(*) from beacon_head_state_total_validators_total
name: beacon_head_state_total_validators_total
time count_gauge
---- -----------
1970-01-01T00:00:00Z 7
```

Other metrics collected by the same telegraf (cpu, load average, teperatures, etc.) are stored just fine, same for metrics reported directly to influxdb by other clients on the same machine (geth).

__Environment info:__

* System info: server Linux armv7l, telegraf Linux x86_64
* InfluxDB version: InfluxDB v1.8.3 (git: 1.8 563e6c3d1a7a2790763c6289501095dbec19244e)
* Other relevant environment details: influxdb / telegraf installed from the ubuntu apt repo on both machines

__Config:__
Commented and empty lines stripped:
```
reporting-disabled = true
[meta]
dir = "/var/lib/influxdb/meta"
retention-autocreate = true
logging-enabled = true
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
cache-max-memory-size = "64m"
series-id-set-cache-size = 100
[coordinator]
[retention]
enabled = true
[shard-precreation]
[monitor]
[http]
[logging]
level = "info"
[subscriber]
[[graphite]]
enabled = true
database = "graphite"
bind-address = ":2003"
protocol = "tcp"
[[collectd]]
[[opentsdb]]
[[udp]]
[continuous_queries]
[tls]
```

Contributor guide

Open the contributing guide

Research direction

The report names no source file or test; start by reproducing the Prometheus-to-InfluxDB path with the supplied configuration and compare scraped series with the resulting query counts. Done means the reported metric points persist like the other Telegraf and geth metrics, with the cause of any loss identified and covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
databases, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.