google / google/weather-tools

Support data ingestion from the `BUFR` format.

Open
#196 0 comments 0 reactions 0 assignees View on GitHub
P1 weather-mv
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

To serve internal data needs, and to more generally support handling of weather data, let's extend the weather-mv to ingest data in the [BUFR](https://en.wikipedia.org/wiki/BUFR) format.

At this stage, the mover supports multiple targets. This feature should enable e2e ingestion of data into BigQuery and Earth Engine. While MetView supports [reading BUFR data](https://metview.readthedocs.io/en/latest/data_types/bufr.html#Bufr), it does not include regridding capability. Thus, `weather-mv rg` does not need need BURF support.

## Implementation Notes:
* Recommended library: https://github.com/ecmwf/pdbufr
* Instead of loading data into an XArray Dataset, pdbufr loads data into a Pandas Dataframe. Our pipeline will have to accommodate this change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.