prometheus / prometheus/statsd_exporter
[breaking] Reduce allocations in `LineToEvents` by returning a single event instead of an slice
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 987
- Forks
- 259
- Avg merge
- 22h 10m
- Merged PRs (30d)
- 2
Description
Description
The current implementation of the LineToEvents method in pkg/line/line.go returns multiple events, which results in a lot of allocations in the statsd-exporter. We propose changing this method to return a single event instead.
On the same refactoring step, we should introduce a "exploder" or "sampling" layer (or helper method on the event) which will keep the same behavior as we have now.
This is a breaking change on purpose which will force client projects using this repository as a library, to adapt to the new method signature.
Here is an example profile with samples using sampling rate, where you can see it takes more than 21% of total memory used:
Goals
- Reduce Allocations: By returning a single event, we aim to reduce the number of allocations and improve performance.
- Enforce Build Breaks: Changing the method signature will force dependent projects to update their code, ensuring compatibility with the new implementation.
- Introduce New Layer: Push the decision about what to do with
CountorSamplingRateto another layer of the exporter.
Proposed Solution
- Change Method Signature: Modify the
LineToEventsmethod to return a single event. - Update Event Structure: Include
CountorSamplingRatein the event structure. - Introduce New Layer: Implement a new layer in the exporter to handle the decision-making process for
CountorSamplingRate.
Repro script
dogClient, err := statsd.New(
opts.statsdServer,
statsd.WithoutTelemetry(),
statsd.WithNamespace("flood_statsd"),
statsd.WithTags([]string{"pod_name:" + os.Getenv("POD_NAME")}),
statsd.WithoutClientSideAggregation(),
)
if err != nil {
level.Error(logger).Log("msg", "Error creating dogClient client", "err", err)
os.Exit(1)
}
samplesSent := 0
start := time.Now()
for {
randomInt := rand.Int63n(opts.cardinalityLimit)
err := dogClient.Count("sample_counter", 1, []string{"mybad_label:co_" + strconv.FormatInt(randomInt, 10)}, 0.01)
if err != nil {
level.Error(logger).Log("msg", "Error sending metric", "err", err)
}
_ = dogClient.Distribution("synthetic", float64(randomInt), []string{"mybad_label:co" + strconv.FormatInt(randomInt, 10)}, 0.001)
_ = dogClient.Timing(
"some_timing",
time.Duration(rand.Intn(3000))*time.Millisecond,
[]string{"cardinality:" + strconv.FormatInt(randomInt, 10)},
0.01,
)
_ = dogClient.Gauge("some_gauge", float64(randomInt), []string{"mybad_label:co" + strconv.FormatInt(randomInt, 10)}, 0.01)
_ = dogClient.Distribution("heavily_sampled_distribution", float64(randomInt), []string{"mybad_label:co" + strconv.FormatInt(randomInt, 10)}, 0.001)
// control the rate of the synthetic metrics
samplesSent += 4
if samplesSent >= int(opts.samplesPerSec) {
elapsed := time.Since(start)
if elapsed < time.Second {
level.Info(logger).Log("msg", "Sleeping for", "duration", time.Second-elapsed)
time.Sleep(time.Second - elapsed)
}
level.Info(logger).Log("msg", "Sending", "samples", samplesSent, "elapsed", elapsed)
samplesSent = 0
start = time.Now()
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/line/line.go at LineToEvents and review the repro script and allocation profile described in the issue. Determine how Count and SamplingRate should be represented in a single event and where the new sampling or exploder layer belongs. Done means the signature changes, existing behavior is preserved, dependent uses are adapted, and the allocation improvement is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100