openclimatefix / openclimatefix/ocf-data-sampler
Add flag for when pv generation is NaN
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 43
- PR merge metrics
- No merged PRs in 30d
Description
Currently in the sites pipeline, when there is no generation data for sites, we infill any NaNs with 0's.
This as @JackKelly has stated is quite "mean" to the model, expecting it know when we don't have site generation data but are passing it 0 values.
One solution to this could be to change the value from 0, to a very specific number that the model could learn that when site generation is that value that there is no real generation.
Another solution which I propose in this issue (thank you @Sukh-P) for the idea, is to pass a flag to the model for each site generation for when it is available. This could be an extra datavariable which is 1, for when site generaiton exists and 0 for when we have replaced NaNs in the dataset for 0. This hopefully allow the model to learn when there is No PV generation better.
Open to thoughts on other solutions and how this could be implemented.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the sites pipeline and the step that replaces missing generation values with zero. Review how site-generation variables are passed to the model, then determine where the availability flag should be added and how it should behave for observed versus replaced values. Done means the pipeline supplies the flag consistently and coverage is validated with relevant tests or sample data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100