prometheus / prometheus/common

MetricFamilyToOpenMetrics produces output that cannot be parsed

Open
#319 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
367
Avg merge
2d 10h
Merged PRs (30d)
18

Description

I am debugging https://github.com/prometheus/jmx_exporter/issues/621, and I found I can reproduce the root cause of the error with the following test in github.com/prometheus/common/expfmt:

package expfmt

import (
	"bytes"
	"strings"
	"testing"
)

func TestJvmClassesLoaded(t *testing.T) {
	in := `
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 6129.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 6143.0
`
	metricFamilies, err := parser.TextToMetricFamilies(strings.NewReader(in))
	if err != nil {
		t.Error(err)
	}
	openMetricsOutput := bytes.NewBuffer(make([]byte, 0))
	for _, metricFamily := range metricFamilies {
		if _, err := MetricFamilyToOpenMetrics(openMetricsOutput, metricFamily); err != nil {
			t.Fatal(err)
		}
	}
	if _, err := FinalizeOpenMetrics(openMetricsOutput); err != nil {
		t.Fatal(err)
	}
	_, err = parser.TextToMetricFamilies(strings.NewReader(openMetricsOutput.String()))
	if err != nil {
		t.Error(err)
	}
}

It fails with

=== RUN   TestJvmClassesLoaded
    jvm_classes_loaded_test.go:33: text format parsing error in line 4: second HELP line for metric name "jvm_classes_loaded"
--- FAIL: TestJvmClassesLoaded (0.00s)
FAIL
FAIL    github.com/prometheus/common/expfmt     0.002s
FAIL

As the error message indicates, the reason is that the _total suffix is truncated in HELP and TYPE (as defined in #214), and the resulting "short name" conflicts with the name of the gauge.

Two full serialization steps are needed to trigger the error (OpenMetrics format -> Go objects -> OpenMetrics format -> Go objects), so this might seem like an issue that doesn't occur in practice. However, in https://github.com/prometheus/jmx_exporter/issues/621 this happens because of the following setup:

jmx_exporter -> exporter_exporter -> Prometheus server

I'm not sure what to do about this. Should exporters prevent metrics from having the same "short name"?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the expfmt package with MetricFamilyToOpenMetrics, FinalizeOpenMetrics, and parser.TextToMetricFamilies, using the reproducer in the issue as the first test to run. Trace how the _total suffix is handled during serialization and parsing. Done means the reported OpenMetrics round trip no longer produces the conflicting HELP error, with the handling of duplicate short names clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.