fsspec / fsspec/kerchunk

test_extract_datatree_chunk_index fails with eccodes v2.38.0

Open
#508 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

As first noticed in https://github.com/fsspec/kerchunk/pull/506, a test is failing when eccodes v2.38.0 is installed.

Observed in CI:

>       assert (
            idx_df["typeOfLevel"][[42, 46, 49, 50]]
            == ["heightAboveGround", "isobaricInhPa", "surface", "heightAboveGround"]
        ).all()
E       AssertionError: assert False
E        +  where False = <bound method NDFrame._add_numeric_operations.<locals>.all of 42     True\n46     True\n49     True\n50    False\nName: typeOfLevel, dtype: bool>()
E        +    where <bound method NDFrame._add_numeric_operations.<locals>.all of 42     True\n46     True\n49     True\n50    False\nName: typeOfLevel, dtype: bool> = 42    heightA... dtype: object == ['heightAbove...tAboveGround']
E             Full diff:
E             - [
E             -  'heightAboveGround',
E             ? ^^                 --
E             + 42    heightAboveGround
E             ? ^^^^^^
E             -  'isobaricInhPa',...
E
E             ...Full output truncated (8 lines hidden), use '-vv' to show.all

Troubleshooting by @maresb:

idx_df.iloc[83]
varname                                                             wz
typeOfLevel                                              isobaricInhPa
stepType                                                       instant
name                                       Geometric vertical velocity
isobaricInhPa                                                    975.0
step                                                   0 days 06:00:00
time                                               2023-09-28 00:00:00
valid_time                                         2023-09-28 06:00:00
uri                            ~/repos/kerchunk/kerchunk/tests/gfs....
offset                                                        21234675
length                                                         1035696
inline_value                                                      None
surface                                                            NaN
heightAboveGround                                                  NaN
meanSea                                                            NaN
Name: 83, dtype: object

From @Anu-Ra-g with eccodes v2.36.0:

varname                                                           watr
typeOfLevel                                                    surface
stepType                                                         accum
name                                                      Water runoff
isobaricInhPa                                                      NaN
step                                                   0 days 06:00:00
time                                               2023-09-28 00:00:00
valid_time                                         2023-09-28 06:00:00
uri                  ./kerchunk/tests/gfs.t00z.pgrb2.0p25.f006.test...
offset                                                        59215098
length                                                          348767
inline_value                                                      None
surface                                                            0.0
heightAboveGround                                                  NaN
meanSea                                                            NaN
Name: 83, dtype: object

Even though the error is occuring, the remaining steps work fine. image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running test_extract_datatree_chunk_index with eccodes v2.38.0 and inspect the reported rows, especially indices 42, 46, 49, and 50. Compare the extracted typeOfLevel values with the eccodes v2.36.0 output and verify that the test and subsequent extraction steps pass consistently across the affected version.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.