Deltares / Deltares/imod-python
Refactor time handling utilities now that we have pandas 3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 12
- Avg merge
- 21h 8m
- Merged PRs (30d)
- 1
Description
Pandas 2 takes nanosecond by default as its base for datetimes. This means that the years that can stored in this format are limited between 1678 and 2261. We ran into these limits more often than one would expect beforehand, and thus have spent quite some time in the past adding functionality to work around this, to check for this, or to deal with this. For example, including support for cftime.
Pandas 3 uses microsecond by default as its base for datetimes, which has a year range roughly inbetween ~290,000 BC to ~290,000 AD. iMOD5 supports years ranging from 1900 AD up to 9999 AD, so well within limits of what is supported by a microseconds base. For paleo-reconstructions where million years are simulated, this still isn't enough, though I don't foresee that usecase any time soon. In short: Our lives are made a lot easier now.
If we drop support of pandas 2, we can simplify the codebase as follows:
- the calls to
_check_yearneed to be removed. These now block users from setting up simulations beyond the year 2261 with pandas 3. - support for
use_cftimecan be dropped, reducing quite some if...else clauses
Before we drop this stuff, we will first enter a transition phase where the following is required:
- add some logic to
_check_yearthat accounts for the base of the datetime that is used, so that users using pandas 3.0 don't run into errors and throws a warning in case pandas 2 is used that behavior will change. - throw a DeprecationWarning whenever
use_cftimeis used and some instructions how to migrate to pandas 3.0
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the _check_year calls and the code paths using use_cftime. First inspect how datetime bases and pandas versions are currently detected, then trace the related time-handling utilities. Done means pandas 3 users are not blocked by the old year limit, pandas 2 users receive a warning about the changed behavior, and use_cftime emits a DeprecationWarning with migration guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100