open-telemetry / open-telemetry/opentelemetry-java

Benchmark performance report published (and repeatable)

Open
#2,577 0 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature Request help wanted
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Is your feature request related to a problem? Please describe.
I'd like to very quickly be able to find performance characteristics of SDK components. The reasons are severalfold:

  • I'd like to understand if the Google-Cloud exporters are on-par with OTLP performance
  • I'd like to understand the CPU/Memory pressure caused via OTel's SDK and various mechanisms to alleviate it
  • I'd like to understand if we should be promoting different defaults for different envirornments, e.g. K8s vs. K-Native may be highly different in terms of what recommended setup we should promote.

Those are just top-of-mind. In general, it'd be nice if performance characteristics were as public as posisble (without requiring one to run the entire suite).

Describe the solution you'd like
I'd like to update the JMH hooks in the gradle build with some kind of task that will collect and generate markdown reports on that state of any given PR.

The goal would be to (one day) extend the performance monitoring-github action to autopublish performance dumps and over-time analysis of this SDK.

However, I see it as a multi-step process:

  • Create a task which pulls all JMH reports into one central report dump (can be multiple markdown files.
  • Enhance the task in a series of PRs and discussions until we're happy with its format.
  • Automate a GithubAction to generate performance reports when commits hit master (or every so often).
    Note: The Python SDK already does this, and it's pretty nice.

Describe alternatives you've considered
Keeping things as is. Didn't really consider many alternatives so happy to hear ideas.

Additional context
Python configuration: https://github.com/open-telemetry/opentelemetry-python/blob/master/.github/workflows/test.yml#L72
Python example: https://open-telemetry.github.io/opentelemetry-python/benchmarks/

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the JMH hooks in the Gradle build and review the linked Python workflow and benchmark example for report-generation patterns. The first milestone is a task that gathers JMH reports into central Markdown files; the broader work also calls for format iteration and a GitHub Action for publishing reports.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, java
Domain
build-system, ci-cd, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.