[Feature Request]: eagerly evaluating pipeline starting with Create() in Python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
For documentation and testing, I find it super convenient to be able to pass Python objects such as a list into a PTransform, which causes the PTransform to be evaluated eagerly, e.g.,
```
>>> [1, 2, 3] | beam.Map(lambda x: x ** 2)
[1, 4, 9]
```
I would like to be able to do the same sort of thing for pipelines that start with `beam.Create()` rather than an existing PCollection. This would be particularly helpful for examples where I use `beam.Create()` inside my library's own custom PTransforms (and thus cannot simply unwrapped `beam.Create`).
This would need to use some sort of special placeholder value, e.g., perhaps `None` or another sentinel value/class, e.g., perhaps
```
>>> beam.EagerPipeline() | beam.Create([1, 2, 3]) | beam.Map(lambda x: x ** 2)
[1, 4, 9]
```
To solve this problem inside Xarray-Beam (for unit-testing), I wrote a funny EagerPipeline class that actually serializes the results to a temporary file on disk:
https://github.com/google/xarray-beam/blob/037d1f3ea077a9d97e614dd4ac2959ee45633df9/xarray_beam/_src/test_util.py#L32-L51
I guess I could externalize this to my users but I would love to be able to point to something upstream.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.