apache / apache/beam

Fully support dropwizard metrics in beam and runners

Open
#18,470 1 comment 0 reactions 0 assignees View on GitHub
core flink improvement java P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

As proposed at https://docs.google.com/document/d/1-35iyCIJ9P4EQONlakgXBFRGUYoOLanq2Uf2sw5EjJw/edit?usp=sharing , I'd like to add full support of dropwizard metrics by delegating beam metrics to runners.

The proposal involves a few subtasks, as far as I see, including:
1. add Meter interface in sdk-java-core and extend Distribution to support quantiles
2. add MeterData, extend DistributionData. Merge {Counter/Meter/Gauge/Distribution}Data instead of Counter/Meter/Gauge/Distribution at accumulators.
3. Runner changes over improved metrics.

I will create subtasks later if there's no objection.

Imported from Jira [BEAM-2602](https://issues.apache.org/jira/browse/BEAM-2602). Original Jira may contain additional context.
Reported by: Cody.

Contributor guide

Open the contributing guide

Research direction

Start with the linked proposal, then inspect sdk-java-core and the named Meter, Distribution, MeterData, and DistributionData components, along with accumulator and runner code. Done means the listed metric types are supported through the SDK, accumulators, and runners, including quantiles and runner delegation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.