influxdata / influxdata/influxdb
prometheus measurements not being recorded
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. configure telegraf to scrape some prometheus endpoints (in this case, lighthouse beacon and validator)
2. attempt to graph them in chronograf
__Expected behavior:__
Data gets recorded, graphs show things
__Actual behavior:__
Data gets scraped by telegraf, gets submitted to influxdb (confirmed by tcpdump and http `write-tracing` in `influxdb.conf`), but only the first few data points seem to make it to disk; there is no error message, things just get dropped silently (this has been collecting data for 6 days, there should be way more points):
```
name: beacon_head_state_total_validators_total
time gauge host url
---- ----- ---- ---
2020-12-09T17:01:10Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:20Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:30Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:40Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:01:50Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:02:00Z 39885 eth2v1 http://127.0.0.1:5054/metrics
2020-12-09T17:02:10Z 39885 eth2v1 http://127.0.0.1:5054/metrics
> select count(*) from beacon_head_state_total_validators_total
name: beacon_head_state_total_validators_total
time count_gauge
---- -----------
1970-01-01T00:00:00Z 7
```
Other metrics collected by the same telegraf (cpu, load average, teperatures, etc.) are stored just fine, same for metrics reported directly to influxdb by other clients on the same machine (geth).
__Environment info:__
* System info: server Linux armv7l, telegraf Linux x86_64
* InfluxDB version: InfluxDB v1.8.3 (git: 1.8 563e6c3d1a7a2790763c6289501095dbec19244e)
* Other relevant environment details: influxdb / telegraf installed from the ubuntu apt repo on both machines
__Config:__
Commented and empty lines stripped:
```
reporting-disabled = true
[meta]
dir = "/var/lib/influxdb/meta"
retention-autocreate = true
logging-enabled = true
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
cache-max-memory-size = "64m"
series-id-set-cache-size = 100
[coordinator]
[retention]
enabled = true
[shard-precreation]
[monitor]
[http]
[logging]
level = "info"
[subscriber]
[[graphite]]
enabled = true
database = "graphite"
bind-address = ":2003"
protocol = "tcp"
[[collectd]]
[[opentsdb]]
[[udp]]
[continuous_queries]
[tls]
```
Contributor guide
Research direction
The report names no source file or test; start by reproducing the Prometheus-to-InfluxDB path with the supplied configuration and compare scraped series with the resulting query counts. Done means the reported metric points persist like the other Telegraf and geth metrics, with the cause of any loss identified and covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus
- Domain
- databases, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100