[Bug]: Decreased Dataflow runner functionality due to feature availability checks
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Scenario:
- Java
- using Dataflow
- 2.45.0
- streaming
- streaming engine disabled
- using `GroupIntoBatches.WithShardedKey` (for example `BigQueryIO.Write.withAutoSharding`)
The pipeline fails saying it requires Streaming Engine. Based on my knowledge it should work without Streaming Engine as well.
1. Using `GroupIntoBatches.WithShardedKey` triggers https://github.com/apache/beam/blob/v2.45.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L554-L558
2. The referenced `StreamingGroupIntoBatchesWithShardedKeyOverrideFactory` replaces the transform with `StreamingGroupIntoBatchesWithShardedKey`
3. Inside that it calls `DataflowRunner.maybeRecordPCollectionWithAutoSharding` https://github.com/apache/beam/blob/v2.45.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/GroupIntoBatchesOverride.java#L335
4. ... and finally that runner method contains the code that always fails if Streaming Engine isn't enabled. https://github.com/apache/beam/blob/v2.45.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L1697-L1706
I don't think that check is necessary. We already have a `DataflowRunner.verifyDoFnSupported` that should cover checking if the used state functionality is supported or not.
https://github.com/apache/beam/blob/v2.45.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L2442-L2487
I can take this issue and contribute a fix if my assumption is right. I personally can't really see the reasoning behind that limitation (I also checked the original PR https://github.com/apache/beam/pull/13208) - but maybe I missed something.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.