lightninglabs / lightninglabs/lndmon

Performance issue with /metrics endpoint

Open
#28 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
160
Forks
53
PR merge metrics
No merged PRs in 30d

Description

I am trying to use lnd+lndmon on a rock64 board (similar to rpi, with arm64 and 4GB RAM) but Grafana only shows data points coming directly from lnd (Go Runtime + Performance dashboard). Everything supposed to come from lndmon is not there.

I noticed that when running simple queryes with PromQL I immediately got the error: "the queries returned no data for a table". Then went to Explore section and checked for `up`, there I can see how the lndmon process is reported to be down, which is not true.

After that I tried to get the metrics directly and I realized I was getting slow response times on the metrics endpoint (between 10s and 12s usually):

```
$ time curl -s --output /dev/null localhost:9092/metrics

real 0m10.717s
user 0m0.022s
sys 0m0.015s
```

I haven't investigated this deeply yet but the instance has more than enough Ram, and the CPU usage and load average don't look that bad.

Will try to spend more time in another moment but wanted to report soon just in case it's happening to more people.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported 10–12 second response from the /metrics endpoint with curl and compare it with the lndmon process status shown in Prometheus. Trace the endpoint's request path and metric collection behavior; done means lndmon metrics are exposed reliably, the process is reported up, and the Grafana data is available without the observed delay.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grafana, prometheus
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.