Faster ingestion into BQ by converting the chunk into pd.Dataframe
Open
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
In `weather-mv` we're dividing the Dataset into small chunks that's adding appropriate parallelism in the pipeline. In the next step if we convert those small chunks into pandas Dataframes it would reduce the cost of generating the flat rows as extracting the rows from Dataframe is very fast.
```
df = ds.to_dataframe().reset_index()
```
Here `ds` is a small chunk of a dataset and `reset_index()` will flatten the dataset chunk into a normalized dataframe.
Contributor guide
Assessment
This issue has not been assessed yet.