Speed up BigQuery ingestion with Splittable DoFns
Open
P1
weather-mv
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
Quick notes:
- SDFs will allow us to maximize worker utilization in processing weather data (e.g. gribs) in a Dataflow-aware way
- By paying the penalty of opening the same file multiple times, we can process corpuses of data much faster.
- https://beam.apache.org/blog/splittable-do-fn/ is a useful explainer for SDFs, apart from the general programming guide.
Contributor guide
Assessment
This issue has not been assessed yet.