AbstractGlobalCombineFn hierarchy is inconsistent
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Subclasses of AbstractGlobalCombineFn seem to be arranged in a way that prevents them from being used with the DataflowRunner.
Subclasses of AbstractGlobalCombineFn are under either CombineFn or CombineFnWithContext, which seems to be in itself a CombineFn which has access to PipelineOptions and Side Inputs.
However, the DataflowRunner casts all combiners passed from user code to CombineFn (see [1]) which prevents combiners that extend CombineFnWithContext from being used there.
For example:
public class CustomCombinerFn extends CombineWithContext.CombineFnWithContext<...\> \{...}
final PCollectionView\> newCollection = oldCollection
.apply("Custom Combiner", Combine.globally(new CustomCombinerFn(filter))
.withSideInputs(filter)
.withoutDefaults()
.asSingletonView());
IMHO either CombineFnWithContext should be a subclass of CombineFn or DataflowRunner should cast the combiner to AbstractGlobalCombineFn.
[1] https://github.com/apache/beam/blob/b83b302ef97767e4ca245ea24e8bd40a6692e72c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L514
Imported from Jira [BEAM-6511](https://issues.apache.org/jira/browse/BEAM-6511). Original Jira may contain additional context.
Reported by: nlofeudo.
Contributor guide
Research direction
Start with runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java at the referenced cast near line 514, then inspect the AbstractGlobalCombineFn, CombineFn, and CombineFnWithContext hierarchy. Determine which hierarchy change is intended and verify that a context-aware combiner using side inputs can run with DataflowRunner without the cast failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100