4paradigm / 4paradigm/OpenMLDB

monitor issues

Offen
#3,822 1 Kommentar 0 Reaktionen 1 zugewiesene Person Beansprucht von @vagetablechicken Auf GitHub ansehen
monitoring
Vorherrschende Sprache
C++
Sterne
1.7k
Forks
331
Ø Merge
12 T. 12 Std.
Gemergte PRs (30 T.)
1

Beschreibung

origin https://github.com/vagetablechicken/fedb/issues/13

- taskmanager has no prometheus api, so don't add it in prometheus config
- / or /status print a html page, not good for prometheus, maybe openmldb-exporter can handle it, but info is simple.

- what means openmldb_connected_seconds_total? https://github.com/4paradigm/openmldb-exporter/blob/53792409f9a53cceaacf0091f658ca124692d356/openmldb_exporter/collector/collectors.py#L80
`current time - connect time in show component` incorrect: when openmldb-exporter is alive, stop clear start all, it can't reconnect to zk, and fail to execute show components or table status, won't update counters, so the openmldb_connected_seconds_total won't change. only zk restart can be handled, reset(start from small zxid) can't, check if needs improve
why grafana var role doesn't have tm? fix grafana dashboard template

- Table Partition Number show deleted tables partitions, we know it's deleted in graph, but the big number make me confused.
image

- openmldb_table_rows will up and down when insert data
image
test case is insert data into disk table

- two panel are the same metric, but different titile, confused
image

TODO all metrics in grafana, and how to read them when do perf profiling

# About doc
It's better to give a simple and clear steps for monitor deployment, current doc is hard to read.

## insert

When we want to know insert perf, we can get total time by client who do insertion.
DOING: kafka->openmldb perf test in openmldb-compose

# Grafana Update
- [ ] show taskmanager status in first row
- [ ] put metrics of tablets: we can know the insertion performance `xxx_openmldb_api_tablet_server_put[_xxx]`, latency and qps

new grafana dashboard:
https://gist.github.com/vagetablechicken/9428268b8cb8ba50d5867e6f0326da64

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue points to openmldb-exporter collector code and Grafana dashboards. Start by examining the exporter's collector/collectors.py to understand the connected_seconds_total metric logic. Review the linked Grafana gist for dashboard updates. Testing requires a running OpenMLDB cluster with the exporter to reproduce the metric issues and verify fixes.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
grafana, prometheus, python
Bereich
devops, documentation, observability
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.