openclimatefix / openclimatefix/ocf-data-sampler
Bug in t0 embedding when using hour periods and linear embedding type
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 43
- PR merge metrics
- No merged PRs in 30d
Description
There is a bug in the get_t0_embedding() function in this line.
The erroneous line reads:
frac = (t0.hour + t0.minute / 60) / period_hours
But it should instead read something like:
frac = (t0.hour % period_hours + t0.minute / 60) / period_hours
A corresponding bug is in the pytests here
The mistake is only made for hourly configurations, the year fraction configuration has the required modulo operatior.
The mistake does not impact the cyclic embedding since we plug frac into cos and sin functions and sin(2*pi*frac)==sin(2*pi*(frac+N) where N is a whole number. However, when we use a linear embedding the output is incorrect. See the figure below.
In the figure below the "uncorrected" version is the version as in the library today. The "corrected" version is with the line above substituted in.
These 3 plots correspond to using ("1h", "linear"), ("2h", "linear"), and ("6h", "linear") in the embedding function.
What makes this complicated and why I haven't jumped to fix it is that correcting it will break backwards compatibility with models which have be trained with the incorrect version as in the library today.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ocf_data_sampler/numpy_sample/datetime_features.py at get_t0_embedding(), then inspect tests/numpy_sample/test_datetime_features.py around line 36. Run the relevant pytest tests and update the hourly linear-embedding expectations so they cover the reported modulo behavior; completion also requires resolving the stated model backward-compatibility concern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100