NatLabRockies / NatLabRockies/rdtools

YoY double-counting/asymmetry bug for leap days

Open
#283 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
189
Forks
82
PR merge metrics
No merged PRs in 30d

Description

While playing around with #282 I noticed an asymmetry in how leap days contribute to the distribution of YoY slopes. Ignoring filtering and first/last year complications for a moment, each day in the aggregated series is supposed to contribute to one forward and one backward slope. However it seems that there is a small bug related to leap days where a single point can contribute to three slopes instead of two.

To reproduce:

import pandas as pd
import rdtools
import matplotlib.pyplot as plt
daily_pm = pd.Series(1, index=pd.date_range('2014-01-01', '2017-12-31', freq='d'))
daily_pm.loc['2015-02-28'] = 0  # outlier point that interacts with a leap day
rd, ci, calc_info = rdtools.degradation.degradation_year_on_year(daily_pm)

fig = rdtools.plotting.degradation_summary_plots(rd, ci, calc_info, daily_pm)
fig.axes[1].set_ylim(0, 10)  # shrink y-axis to show detail

image

Note that in the histogram plot, the left-most bin has height=1 and the right-most bin has height=2. So a single outlier day that interacts with a leap day creates one big negative slope but two big positive slopes. Examining the df variable inside rdtools.degradation.degradation_year_on_year confirms this -- Feb 28 gets paired with Feb 28, but it also gets paired with Feb 29 in one direction:

df.loc['2015-02'].tail():

                   dt  energy   dt_right  energy_right dt_shifted  time_diff_years    yoy
dt
2015-02-24 2015-02-24     1.0 2014-02-24           1.0 2015-02-24              1.0    0.0
2015-02-25 2015-02-25     1.0 2014-02-25           1.0 2015-02-25              1.0    0.0
2015-02-26 2015-02-26     1.0 2014-02-26           1.0 2015-02-26              1.0    0.0
2015-02-27 2015-02-27     1.0 2014-02-27           1.0 2015-02-27              1.0    0.0
2015-02-28 2015-02-28     0.0 2014-02-28           1.0 2015-02-28              1.0 -100.0

df.loc['2016-02'].tail():

                   dt  energy   dt_right  energy_right dt_shifted  time_diff_years         yoy
dt
2016-02-25 2016-02-25     1.0 2015-02-25           1.0 2016-02-25          1.00000    0.000000
2016-02-26 2016-02-26     1.0 2015-02-26           1.0 2016-02-26          1.00000    0.000000
2016-02-27 2016-02-27     1.0 2015-02-27           1.0 2016-02-27          1.00000    0.000000
2016-02-28 2016-02-28     1.0 2015-02-28           0.0 2016-02-28          1.00000  100.000000
2016-02-29 2016-02-29     1.0 2015-02-28           0.0 2016-02-28          1.00274   99.726776

I suspect, but did not verify, that this has to do with pd.merge_asof's default choice of direction='backward'. Possible solutions:

  • do nothing because this probably has negligible impact on results
  • filter out leap days from the series before calculating YoY slopes
  • something else?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pandas reproduction and inspect rdtools.degradation.degradation_year_on_year, focusing on how the YoY pairs are formed around February 28 and 29. Verify the resulting slope distribution and determine how the leap-day pairing should be handled. Done means a single outlier contributes once in each intended direction rather than being counted an extra time.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.