intake / intake/intake-esm

ENH: Searching within a time interval

Open
#623 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

(I first made this as a comment to PR #291, but I realized afterwards that the PR had been staled for three years, so I figured out that a new issue might be more appropriate.)

We had this issue and need when writing [xscen](https://github.com/Ouranosinc/xscen/). Our solution does not yet feel generic enough to be implemented here, but if you want to have a look at a first version is here : https://github.com/Ouranosinc/xscen/blob/b261a04ed73e398a60a0632bdb29be324dc3f5b6/xscen/catalog.py#L899-L998

The idea is similar to the staled PR. You have a "date_start" and a "date_end" column in the catalog. For a given "period", the code returns the row where the `date_start - date_end` interval overlaps with that period. Because of the limitations of pandas <2, we had to use `pd.Period` objects in our catalogs and the code suffers a bit from this workaround.

In addition to a simple "overlap", our function tries to guess the percentage of the period that is covered by the rows of the dataframe, so we only return the subset if a significant percentage is obtained. This has the restriction that it make sense if the rows of the dataset are not temporally overlapping, like for a single variable divided temporally in multiple files. A "full overlap" would often be too strict because of so many caveats (different calendars, imprecise date bounds).

I recently tried to use `datetime64[ms]` columns with pandas >= 2, which allows to simplify the function a bit and use more `pd.Interval` magic. It is here: https://github.com/Ouranosinc/xscen/blob/05054bfbf450c6b332e239e7866f766f51a47ed0/xscen/catalog.py#L892-L974.

There are still some caveats and questions to answer I think.
- How do we tell `intake-esm` which columns are the time bounds ?
- How do we solve the "coverage" issue neatly ?

With input from the intake-esm devs and users, we (Ouranos) could consider investing some time into adapting and upstreaming our solution as we would be more than happy to make `xscen` thinner.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the stalled PR #291 and the two referenced implementations in xscen/catalog.py, especially the interval-overlap and coverage logic. Clarify how intake-esm should identify time-bound columns and how coverage should be handled across calendars and imprecise bounds. Done means the API and behavior are agreed with maintainers and the generic interval search is upstreamed.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.