grib_tree function to properly handle ECMWF ensemble data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
grib_tree function fails to properly handle ECMWF ensemble data
Issue Description
The grib_tree function in kerchunk.grib2 doesn't properly handle ECMWF ensemble forecast data, specifically:
- It fails to recognize and preserve ensemble member information
- It significantly reduces the number of groups in the output compared to input
- It doesn't provide a way to access ensemble dimension in the resulting zarr structure
Reproduction
When processing ECMWF ensemble data with 19 variables and 51 ensemble members (969 total message groups):
from kerchunk.grib2 import scan_grib, grib_tree
import datatree
date_str='20240229'
ecmwf_s3url=f"s3://ecmwf-forecasts/{date_str}/00z/ifs/0p25/enfo/{date_str}000000-0h-enfo-ef.grib2"
esc_groups = scan_grib(ecmwf_s3url)
original_tree = grib_tree(esc_groups)
gfs_dt = datatree.open_datatree(
fsspec.filesystem("reference", fo=original_tree).get_mapper(""),
engine="zarr",
consolidated=False
)
# The key test: can we access ensemble members?
print(gfs_dt.keys()) # Check for variables
The resulting structure loses ensemble information, making it impossible to distinguish between different ensemble members in the output.
This gist explains the situation and a wayforward to have the ensemble number in the grib_tree.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the grib_tree and scan_grib entry points using the ECMWF reproduction in the issue, then read the linked gist for the described way forward. Done means the resulting datatree preserves the 51 ensemble members, retains the expected 969 message groups, and exposes ensemble members for access through the zarr structure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100