openclimatefix / openclimatefix/ocf-data-sampler

Add flag for when pv generation is NaN

Open
#356 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ocf-internal
Dominant language
Python
Stars
20
Forks
43
PR merge metrics
No merged PRs in 30d

Description

Currently in the sites pipeline, when there is no generation data for sites, we infill any NaNs with 0's.

This as @JackKelly has stated is quite "mean" to the model, expecting it know when we don't have site generation data but are passing it 0 values.

One solution to this could be to change the value from 0, to a very specific number that the model could learn that when site generation is that value that there is no real generation.

Another solution which I propose in this issue (thank you @Sukh-P) for the idea, is to pass a flag to the model for each site generation for when it is available. This could be an extra datavariable which is 1, for when site generaiton exists and 0 for when we have replaced NaNs in the dataset for 0. This hopefully allow the model to learn when there is No PV generation better.

Open to thoughts on other solutions and how this could be implemented.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the sites pipeline and the step that replaces missing generation values with zero. Review how site-generation variables are passed to the model, then determine where the availability flag should be added and how it should behave for observed versus replaced values. Done means the pipeline supplies the flag consistently and coverage is validated with relevant tests or sample data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.