MeteoSwiss / MeteoSwiss/probtest

Error handling with stats and files with different dimensions.

Open
#19 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
9
Avg merge
3d 17h
Merged PRs (30d)
3

Description

Currently when using the stats command and a file_id with multiple files, the error thrown is this:
```
Traceback (most recent call last):
File "./probtest.py", line 57, in
cli()
File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/engine/stats.py", line 91, in stats
file_specification,
File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/engine/stats.py", line 15, in create_stats_dataframe
df = df_from_file_ids(file_id, input_dir, file_specification)
File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/util/dataframe_ops.py", line 150, in df_from_file_ids
fid_dfs = unify_time_index(fid_dfs)
File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/util/dataframe_ops.py", line 174, in unify_time_index
df = df.reindex(columns=unique_times, level=time_multiindex_index)
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/util/_decorators.py", line 309, in wrapper
return func(*args, **kwargs)
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/frame.py", line 4036, in reindex
return super().reindex(**kwargs)
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/generic.py", line 4464, in reindex
axes, level, limit, tolerance, method, fill_value, copy
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/frame.py", line 3877, in _reindex_axes
columns, method, copy, level, fill_value, limit, tolerance
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/frame.py", line 3925, in _reindex_columns
allow_dups=False,
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/generic.py", line 4532, in _reindex_with_indexers
copy=copy,
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/internals/managers.py", line 1285, in reindex_indexer
self.axes[axis]._can_reindex(indexer)
File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/indexes/base.py", line 3292, in _can_reindex
raise ValueError("cannot reindex from a duplicate axis")
ValueError: cannot reindex from a duplicate axis
```

We could make a simple check to make sure all files a user is trying to use `stats` on have the same dimensions and give a more meaningful and lightweight error message. E.g: "Error: File a and File b have different dimensions. They must have the same dimensions."

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with engine/stats.py and util/dataframe_ops.py, especially create_stats_dataframe, df_from_file_ids, and unify_time_index from the traceback. Reproduce the stats command with files of different dimensions, then verify it reports which files differ instead of raising pandas' duplicate-axis error.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
cli, data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.