fsspec / fsspec/kerchunk

grib_tree function to properly handle ECMWF ensemble data

Open
#546 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

grib_tree function fails to properly handle ECMWF ensemble data

Issue Description

The grib_tree function in kerchunk.grib2 doesn't properly handle ECMWF ensemble forecast data, specifically:

  1. It fails to recognize and preserve ensemble member information
  2. It significantly reduces the number of groups in the output compared to input
  3. It doesn't provide a way to access ensemble dimension in the resulting zarr structure

Reproduction

When processing ECMWF ensemble data with 19 variables and 51 ensemble members (969 total message groups):

from kerchunk.grib2 import scan_grib, grib_tree
import datatree
date_str='20240229'
ecmwf_s3url=f"s3://ecmwf-forecasts/{date_str}/00z/ifs/0p25/enfo/{date_str}000000-0h-enfo-ef.grib2"
esc_groups = scan_grib(ecmwf_s3url)
original_tree = grib_tree(esc_groups)
gfs_dt = datatree.open_datatree(
    fsspec.filesystem("reference", fo=original_tree).get_mapper(""), 
    engine="zarr", 
    consolidated=False
)

# The key test: can we access ensemble members?
print(gfs_dt.keys())  # Check for variables

The resulting structure loses ensemble information, making it impossible to distinguish between different ensemble members in the output.

This gist explains the situation and a wayforward to have the ensemble number in the grib_tree.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the grib_tree and scan_grib entry points using the ECMWF reproduction in the issue, then read the linked gist for the described way forward. Done means the resulting datatree preserves the 51 ensemble members, retains the expected 969 message groups, and exposes ensemble members for access through the zarr structure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.