4paradigm / 4paradigm/OpenMLDB

monitor issues

Open
#3,822 1 comment 0 reactions 1 assignee Claimed by @vagetablechicken View on GitHub
monitoring
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

origin https://github.com/vagetablechicken/fedb/issues/13

- taskmanager has no prometheus api, so don't add it in prometheus config
- / or /status print a html page, not good for prometheus, maybe openmldb-exporter can handle it, but info is simple.

- what means openmldb_connected_seconds_total? https://github.com/4paradigm/openmldb-exporter/blob/53792409f9a53cceaacf0091f658ca124692d356/openmldb_exporter/collector/collectors.py#L80
`current time - connect time in show component` incorrect: when openmldb-exporter is alive, stop clear start all, it can't reconnect to zk, and fail to execute show components or table status, won't update counters, so the openmldb_connected_seconds_total won't change. only zk restart can be handled, reset(start from small zxid) can't, check if needs improve
why grafana var role doesn't have tm? fix grafana dashboard template

- Table Partition Number show deleted tables partitions, we know it's deleted in graph, but the big number make me confused.
image

- openmldb_table_rows will up and down when insert data
image
test case is insert data into disk table

- two panel are the same metric, but different titile, confused
image

TODO all metrics in grafana, and how to read them when do perf profiling

# About doc
It's better to give a simple and clear steps for monitor deployment, current doc is hard to read.

## insert

When we want to know insert perf, we can get total time by client who do insertion.
DOING: kafka->openmldb perf test in openmldb-compose

# Grafana Update
- [ ] show taskmanager status in first row
- [ ] put metrics of tablets: we can know the insertion performance `xxx_openmldb_api_tablet_server_put[_xxx]`, latency and qps

new grafana dashboard:
https://gist.github.com/vagetablechicken/9428268b8cb8ba50d5867e6f0326da64

Contributor guide

Open the contributing guide

Research direction

The issue points to openmldb-exporter collector code and Grafana dashboards. Start by examining the exporter's collector/collectors.py to understand the connected_seconds_total metric logic. Review the linked Grafana gist for dashboard updates. Testing requires a running OpenMLDB cluster with the exporter to reproduce the metric issues and verify fixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, prometheus, python
Domain
devops, documentation, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.