openclimatefix / openclimatefix/ocf-data-sampler
potentially remove dropna in site find_valid_t0s
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 43
- PR merge metrics
- No merged PRs in 30d
Description
Missed this on merge, but re: comments here:
# 2. Now lets loop over each location in system id and find the valid periods
# Should we have a different option if there are not nans
sites = datasets_dict["site"]
site_ids = sites.site_id.values
site_config = config.input_data.site
valid_t0_and_site_ids = []
for site_id in site_ids:
site = sites.sel(site_id=site_id)
# drop any nan values
# not sure this is right?
site = site.dropna(dim='time_utc')
I don't think we should be doing dropna here. This will make a block of dates with even 1 missing value discontinuous which I think is less beneficial than using it and filling in the missing timestamp later on (e g one missing point can cost you something like 45 potential t0s with 3h history and 8h forecast). I've used data with about 3% missing, sometimes in considerable chunks, and the model seemed to do fine and not get distracted by nan infills.
There is a greater discussion to be had around how much missing data we allow to be infilled and at what times, but I think this should be done in preprocessing anyway and not here; I'd remove it for now.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The relevant entry point is ocf_data_sampler/torch_datasets/site.py, around the site loop and site.dropna(dim='time_utc'); read how valid_t0_and_site_ids is built after this point. Confirm the change preserves timestamps across missing values and avoids discarding potential t0s, then check the resulting sampling behavior against the issue's 3-hour history and 8-hour forecast example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100