google / google/weather-tools

Faster ingestion into BQ by converting the chunk into pd.Dataframe

Open
#414 0 comments 0 reactions 1 assignee Claimed by @DarshanSP19 View on GitHub
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

In `weather-mv` we're dividing the Dataset into small chunks that's adding appropriate parallelism in the pipeline. In the next step if we convert those small chunks into pandas Dataframes it would reduce the cost of generating the flat rows as extracting the rows from Dataframe is very fast.
```
df = ds.to_dataframe().reset_index()
```
Here `ds` is a small chunk of a dataset and `reset_index()` will flatten the dataset chunk into a normalized dataframe.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.