apache / apache/ozhera

[enhancement] Optimization of Metric Calculation in trace-etl-server​

Open
#586 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
122
Forks
46
Avg merge
11h 56m
Merged PRs (30d)
9

Description

Recently, we have further optimized the metric calculation in ​​trace-etl-server​​. The background for this optimization stems from our observation that in today's microservices architecture, there are often applications with extremely high call volumes and numerous upstream/downstream dependencies—we refer to these as **​​"head applications"​​**.

Due to their high traffic, instances of these head applications typically rank among the ​​top consumers​​ in the entire cluster. Additionally, because of their extensive dependencies, the ​​dispersion of metric labels​​ (e.g., Http or Dubbo methodName) can be ​​orders of magnitude higher​​ compared to other applications.

These two factors lead to ​​trace-etl-server generating an enormous volume of metrics​​ when processing head applications. This excessive metric load can cause:

​​**Memory exhaustion**​​ in trace-etl-server itself.
​​**Significant delays**​​ when Prometheus scrapes these metrics.
**​​High pressure**​​ on Prometheus during metric aggregation.

To address these issues, we have implemented the following improvements:

​​1. Configurable metric merging per application.​​
2. ​​A recommendation to isolate head applications​​ by consuming their traces in dedicated ​​trace-etl-server instances​​ in production environments.

This optimization ensures better stability and performance for both ​​trace-etl-server​​ and ​​Prometheus​​ when handling high-traffic services.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.