apache / apache/beam

Pipeline proto seems to be incorrect for Combine.GroupedValues

Open
#18,585 0 comments 0 reactions 0 assignees View on GitHub
core dataflow java P3 portability runners sub-task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

It looks like CombineTest$BasicTests#testHotKeyCombining on Dataflow (and possibly other runners) is creating an invalid pipeline proto since the transform doesn't an environment (and possible a spec):
```

I0610 16:05:23.791430 14054 fnapi_instruction_graph_rewriter.cc:230] transforms {
I0610 16:05:23.791402
14054 fnapi_instruction_graph_rewriter.cc:230] key: "HotMean/PostCombine/Combine.GroupedValues"
I0610
16:05:23.791404 14054 fnapi_instruction_graph_rewriter.cc:230] value {
I0610 16:05:23.791406
14054 fnapi_instruction_graph_rewriter.cc:230] inputs {
I0610 16:05:23.791408 14054 fnapi_instruction_graph_rewriter.cc:230]
key: "org.apache.beam.sdk.values.PCollection.:400#56b99bb29b40d50c"
I0610 16:05:23.791410
14054 fnapi_instruction_graph_rewriter.cc:230] value: "HotMean/PostCombine/GroupByKey.out"
I0610
16:05:23.791412 14054 fnapi_instruction_graph_rewriter.cc:230] }
I0610 16:05:23.791414 14054
fnapi_instruction_graph_rewriter.cc:230] outputs {
I0610 16:05:23.791416 14054 fnapi_instruction_graph_rewriter.cc:230]
key: "org.apache.beam.sdk.values.PCollection.:400#4fa4d31096ca160c"
I0610 16:05:23.791419
14054 fnapi_instruction_graph_rewriter.cc:230] value: "HotMean/PostCombine/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous).output"
I0610
16:05:23.791421 14054 fnapi_instruction_graph_rewriter.cc:230] }
I0610 16:05:23.791423 14054
fnapi_instruction_graph_rewriter.cc:230] unique_name: "HotMean/PostCombine/Combine.GroupedValues"
I0610
16:05:23.791426 14054 fnapi_instruction_graph_rewriter.cc:230] }
I0610 16:05:23.791428 14054
fnapi_instruction_graph_rewriter.cc:230] }
```

Imported from Jira [BEAM-10266](https://issues.apache.org/jira/browse/BEAM-10266). Original Jira may contain additional context.
Reported by: lcwik.
Subtask of issue #18583

Contributor guide

Open the contributing guide

Research direction

Start with CombineTest$BasicTests#testHotKeyCombining and inspect the generated pipeline proto for Combine.GroupedValues, using the reported Dataflow output as the reproduction clue. Determine what environment or spec information is missing and verify that the resulting proto is valid for the affected runners.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.