PCollectionList support in cross-language transforms
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently, Beam model doesn't have any information on the order of input/output PCollections from PTransforms. Therefore, PCollectionList needs to be converted to PCollectionTuple when it goes across the cross-language boundaries (or even in the same language, whenever it is converted between in-memory object and proto) and it's impossible to recreate PCollectionList from proto with the original order. The possible workaround is just to use PCollectionTuple with integer id (starting from 0 like indexes) instead of PCollectionList. In that case, we should first well-define how we generate proto from PCollectionList since each SDK uses a different convention.
Imported from Jira [BEAM-8804](https://issues.apache.org/jira/browse/BEAM-8804). Original Jira may contain additional context.
Reported by: heejong.
Contributor guide
Assessment
This issue has not been assessed yet.