vexxhost / vexxhost/openstack_database_exporter
Wrong openstack_nova_agent_state metric
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 12
- Forks
- 3
- Avg merge
- 31m
- Merged PRs (30d)
- 5
Description
The database exporter does not report the same openstack_nova_agent_state metric as the api exporter.
The API gives you two pieces of information:
- Is admin state enabled or disabled?
- Is the state up or down?
The API exporter puts enabled/disabled as a label and up/down as the value of the metric. If the metric value is 1, it is up.
The database exporter treats them equivalently, which is wrong. This may interfere with important alerts, when the service was enabled but "down" and requires investigation. The database exporter would still report it is up and no alert would be triggered.
In the API code, you find the following: nova/servicegroup/drivers/db.py
This only holds true if nova.CONF.servicegroup_driver=db (which is the default if unset, see docs). If something else is set, this will not work.
Based on this, a potential fix would be:
func computeAgentState(logger *slog.Logger, lastSeenUp, createdAt time.Time, createdAtValid, lastSeenUpValid bool, forcedDown bool, serviceDownTime time.Duration) float64 {
if !createdAtValid {
logger.Warn("Nova service table has faulty values.")
return 0
}
if forcedDown {
return 0
}
heartbeat := createdAt
if lastSeenUpValid {
heartbeat = lastSeenUp
}
elapsed := time.Since(heartbeat)
if elapsed < 0 {
elapsed = -elapsed
}
if elapsed <= serviceDownTime {
return 1
}
return 0
}
agentValue := computeAgentState(c.logger,
service.LastSeenUp.Time,
service.CreatedAt.Time,
service.CreatedAt.Valid,
service.LastSeenUp.Valid,
service.ForcedDown.Valid && service.ForcedDown.Bool,
60)
The current database exporter has no way to change default values besides hardcoding them, so I would also suggest to integrate something that would allow users to pass in their config values via flags. 60s is the default nova.CONF.service_down_time value, see docs
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the API behavior in nova/servicegroup/drivers/db.py, then locate the database exporter’s openstack_nova_agent_state implementation and compare how admin state, service state, and timing are represented. Done means the database exporter matches the API metric semantics and accounts for the configurable service_down_time value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100