openclimatefix / openclimatefix/ocf-data-sampler

Bug in t0 embedding when using hour periods and linear embedding type

Open
#435 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug ocf-internal
Dominant language
Python
Stars
20
Forks
43
PR merge metrics
No merged PRs in 30d

Description

There is a bug in the get_t0_embedding() function in this line.

The erroneous line reads:

frac = (t0.hour + t0.minute / 60) / period_hours

But it should instead read something like:

frac = (t0.hour % period_hours + t0.minute / 60) / period_hours

A corresponding bug is in the pytests here

The mistake is only made for hourly configurations, the year fraction configuration has the required modulo operatior.

The mistake does not impact the cyclic embedding since we plug frac into cos and sin functions and sin(2*pi*frac)==sin(2*pi*(frac+N) where N is a whole number. However, when we use a linear embedding the output is incorrect. See the figure below.

In the figure below the "uncorrected" version is the version as in the library today. The "corrected" version is with the line above substituted in.

These 3 plots correspond to using ("1h", "linear"), ("2h", "linear"), and ("6h", "linear") in the embedding function.

Image

What makes this complicated and why I haven't jumped to fix it is that correcting it will break backwards compatibility with models which have be trained with the incorrect version as in the library today.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ocf_data_sampler/numpy_sample/datetime_features.py at get_t0_embedding(), then inspect tests/numpy_sample/test_datetime_features.py around line 36. Run the relevant pytest tests and update the hourly linear-embedding expectations so they cover the reported modulo behavior; completion also requires resolving the stated model backward-compatibility concern.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.