equinor / equinor/res2df

[summary] df may get duplicated rows when using `time_index=raw`

Open
#426 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
37
Forks
38
Avg merge
1h 45m
Merged PRs (30d)
1

Description

If using `time_index=raw`, the `resample_smry_dates` method is still called to accommodate for functionality like `start_date`, `end_date` and `normalize`, and then used as input when fetching the summary data through `ecl`. If one or more timesteps are shorter than the resolution of the `TIME` vector, resulting in non-unique values in the `TIME` vector, the data is only fetched from the first of the time steps with non-unique `TIME` value giving duplicated rows rather than fetching the data for the updated timestep. This was observed by user after https://github.com/equinor/ecl2df/pull/412, since the first of the non-unique time steps typically has a considerably longer `TIMESTEP` than the subsequent ones, and thus the `TIMESTEP` correction may not be robust due to the possibility of jumping further than the unique `TIME` value.

For `time_index=None` this is not an issue, as `resample_smry_dates` is not called (though that might mean that not all the arguments like `start_date` actually work for `None`?). It seems like we have to call for data from `ecl` for `raw` like we currently do for `None` to make sure that we actually get the `raw` data. If we have to support `start_date`, `end_date` and `normalize`, we likely have to fetch the data twice from `ecl` if they are defined, once for `raw` (like currently for `None`), and then for `start_date`, `end_date` and `normalize`. Then do the possible `TIMESTEP` correction (https://github.com/equinor/ecl2df/pull/412) before cutting/merging the two df's. If so, this should probably be the behavior for both `raw` and `None`

Contributor guide

Open the contributing guide

Research direction

Start at resample_smry_dates and the ecl summary-data fetch, comparing the raw and None paths described in the issue. Reproduce the case with non-unique TIME values and verify that raw data has no duplicated rows while start_date, end_date, normalize, and TIMESTEP correction retain the intended behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.