prometheus / prometheus/client_python

Remove gauge metric

Open
#1,055 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
876
Avg merge
8d 4h
Merged PRs (30d)
1

Description

Hi all,
I have written a custom exporter for calculating the cost of using a node in AWS.
Here is how it works:
Lets assume I have 3 nodes, each costs 5$/ 1h, when I plot using grafana the sum(cost_metric{}) i get 15$ (3x5$). Lets say after 2 hours one of the nodes get deleted (autoscaling). In that case the total cost should drop to 10$.

The problem is that in my case the metric is preserved and even though the node has been deleted the cost is kept and thus it displays 15$ instead of dropping to 10$

How would I go about saving that problem?

cost_metric = Gauge(
    "cost_metric ",
    "Cost of running an instance for 1 hour",
    ["node_name", "instance_type"],
)
...

node_names = get_nodes()
    for node_name in node_names:
        node_info = get_node_info(node_name)
        if node_info is None:
            continue

        logging.info(f"Updating metrics for node: {node_name}")

        # labels section
        labels = node_info["metadata"]["labels"]
        instance_type = labels.get("beta.kubernetes.io/instance-type", "unknown")
        cost = get_cost_of_instance(instance_type)

        if cost is not None:
            cost_metric.labels(node_name=node_name, instance_type=instance_type).set(cost)

I tried

I collected previous and current nodes in form of dict and then wanted to removed the ones that arent existing, the issue is that :

cost_metric.remove(node_name=node_name, instance_type=instance_type)

Traceback (most recent call last):
  File "/home/XXXX/projects/main.py", line 154, in <module>
    previous_nodes = update_metrics(previous_nodes)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eksohio/projects/main.py", line 131, in update_metrics
    cost.remove(node_name=node_name, instance_type=instance_type)
TypeError: MetricWrapperBase.remove() got an unexpected keyword argument 'node_name'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names the Gauge declaration and the MetricWrapperBase.remove call in the traceback, but no repository file or test. Start by checking the Gauge and MetricWrapperBase APIs for label removal and review the exporter loop shown. Done means establishing the supported way to remove the deleted node's metric or confirming that a library change is needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, prometheus, python
Domain
cloud, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.