apache / apache/beam

Complete the removal of aggregators from Spark runner.

Open
#18,425 0 comments 0 reactions 0 assignees View on GitHub
P3 runners spark task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

In https://issues.apache.org/jira/browse/BEAM-1765 aggregator usage was removed from Spark runner in order to comply with the removal of aggregators from the SDK.

However, not all classes / usages of aggregators within the Spark runner were removed.

This task is to complete the removal of all aggregator code from Spark runner, including, but not limited to:

```

org.apache.beam.runners.spark.aggregators.AggAccumParam
org.apache.beam.runners.spark.aggregators.AggregatorsAccumulator
org.apache.beam.runners.spark.aggregators.NamedAggregators
org.apache.beam.runners.spark.metrics.AggregatorMetricSource
org.apache.beam.runners.spark.metrics.AggregatorMetric

```

In addition, remove packages named `aggregators` and move any remaining relevant classes to `org.apache.beam.runners.spark.metrics` package.

Imported from Jira [BEAM-2811](https://issues.apache.org/jira/browse/BEAM-2811). Original Jira may contain additional context.
Reported by: aviemzur.

Contributor guide

Open the contributing guide

Research direction

Start by searching the Spark runner for the listed classes and all references to aggregator code, including the aggregators and metrics packages. Read the BEAM-1765 and BEAM-2811 context before deciding which remaining classes belong under org.apache.beam.runners.spark.metrics. Done means aggregator usages and the aggregators packages are removed, with relevant classes moved to the metrics package.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.