[summary] df may get duplicated rows when using `time_index=raw`
- Dominant language
- Python
- Stars
- 37
- Forks
- 38
- Avg merge
- 1h 45m
- Merged PRs (30d)
- 1
Description
If using `time_index=raw`, the `resample_smry_dates` method is still called to accommodate for functionality like `start_date`, `end_date` and `normalize`, and then used as input when fetching the summary data through `ecl`. If one or more timesteps are shorter than the resolution of the `TIME` vector, resulting in non-unique values in the `TIME` vector, the data is only fetched from the first of the time steps with non-unique `TIME` value giving duplicated rows rather than fetching the data for the updated timestep. This was observed by user after https://github.com/equinor/ecl2df/pull/412, since the first of the non-unique time steps typically has a considerably longer `TIMESTEP` than the subsequent ones, and thus the `TIMESTEP` correction may not be robust due to the possibility of jumping further than the unique `TIME` value.
For `time_index=None` this is not an issue, as `resample_smry_dates` is not called (though that might mean that not all the arguments like `start_date` actually work for `None`?). It seems like we have to call for data from `ecl` for `raw` like we currently do for `None` to make sure that we actually get the `raw` data. If we have to support `start_date`, `end_date` and `normalize`, we likely have to fetch the data twice from `ecl` if they are defined, once for `raw` (like currently for `None`), and then for `start_date`, `end_date` and `normalize`. Then do the possible `TIMESTEP` correction (https://github.com/equinor/ecl2df/pull/412) before cutting/merging the two df's. If so, this should probably be the behavior for both `raw` and `None`
Contributor guide
Research direction
Start at resample_smry_dates and the ecl summary-data fetch, comparing the raw and None paths described in the issue. Reproduce the case with non-unique TIME values and verify that raw data has no duplicated rows while start_date, end_date, normalize, and TIMESTEP correction retain the intended behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100