prometheus / prometheus/statsd_exporter

[breaking] Reduce allocations in `LineToEvents` by returning a single event instead of an slice

Open
#577 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement library
Dominant language
Go
Stars
987
Forks
259
Avg merge
22h 10m
Merged PRs (30d)
2

Description

Description

The current implementation of the LineToEvents method in pkg/line/line.go returns multiple events, which results in a lot of allocations in the statsd-exporter. We propose changing this method to return a single event instead.
On the same refactoring step, we should introduce a "exploder" or "sampling" layer (or helper method on the event) which will keep the same behavior as we have now.
This is a breaking change on purpose which will force client projects using this repository as a library, to adapt to the new method signature.

Here is an example profile with samples using sampling rate, where you can see it takes more than 21% of total memory used:
image

Goals

  1. Reduce Allocations: By returning a single event, we aim to reduce the number of allocations and improve performance.
  2. Enforce Build Breaks: Changing the method signature will force dependent projects to update their code, ensuring compatibility with the new implementation.
  3. Introduce New Layer: Push the decision about what to do with Count or SamplingRate to another layer of the exporter.

Proposed Solution

  1. Change Method Signature: Modify the LineToEvents method to return a single event.
  2. Update Event Structure: Include Count or SamplingRate in the event structure.
  3. Introduce New Layer: Implement a new layer in the exporter to handle the decision-making process for Count or SamplingRate.
Repro script
	dogClient, err := statsd.New(
		opts.statsdServer,
		statsd.WithoutTelemetry(),
		statsd.WithNamespace("flood_statsd"),
		statsd.WithTags([]string{"pod_name:" + os.Getenv("POD_NAME")}),
		statsd.WithoutClientSideAggregation(),
	)
	if err != nil {
		level.Error(logger).Log("msg", "Error creating dogClient client", "err", err)
		os.Exit(1)
	}

	samplesSent := 0

	start := time.Now()
	for {
		randomInt := rand.Int63n(opts.cardinalityLimit)
		err := dogClient.Count("sample_counter", 1, []string{"mybad_label:co_" + strconv.FormatInt(randomInt, 10)}, 0.01)
		if err != nil {
			level.Error(logger).Log("msg", "Error sending metric", "err", err)
		}
		_ = dogClient.Distribution("synthetic", float64(randomInt), []string{"mybad_label:co" + strconv.FormatInt(randomInt, 10)}, 0.001)
		_ = dogClient.Timing(
			"some_timing",
			time.Duration(rand.Intn(3000))*time.Millisecond,
			[]string{"cardinality:" + strconv.FormatInt(randomInt, 10)},
			0.01,
		)
		_ = dogClient.Gauge("some_gauge", float64(randomInt), []string{"mybad_label:co" + strconv.FormatInt(randomInt, 10)}, 0.01)

		_ = dogClient.Distribution("heavily_sampled_distribution", float64(randomInt), []string{"mybad_label:co" + strconv.FormatInt(randomInt, 10)}, 0.001)

		// control the rate of the synthetic metrics
		samplesSent += 4
		if samplesSent >= int(opts.samplesPerSec) {
			elapsed := time.Since(start)
			if elapsed < time.Second {
				level.Info(logger).Log("msg", "Sleeping for", "duration", time.Second-elapsed)
				time.Sleep(time.Second - elapsed)
			}
			level.Info(logger).Log("msg", "Sending", "samples", samplesSent, "elapsed", elapsed)
			samplesSent = 0
			start = time.Now()
		}
	}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/line/line.go at LineToEvents and review the repro script and allocation profile described in the issue. Determine how Count and SamplingRate should be represented in a single event and where the new sampling or exploder layer belongs. Done means the signature changes, existing behavior is preserved, dependent uses are adapted, and the allocation improvement is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.