openclimatefix / openclimatefix/ocf-data-sampler

potentially remove dropna in site find_valid_t0s

Open
#80 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ocf-internal
Dominant language
Python
Stars
20
Forks
43
PR merge metrics
No merged PRs in 30d

Description

Missed this on merge, but re: comments here:

    # 2. Now lets loop over each location in system id and find the valid periods
    # Should we have a different option if there are not nans
    sites = datasets_dict["site"]
    site_ids = sites.site_id.values
    site_config = config.input_data.site
    valid_t0_and_site_ids = []
    for site_id in site_ids:
        site = sites.sel(site_id=site_id)


        # drop any nan values
        # not sure this is right?
        site = site.dropna(dim='time_utc')

I don't think we should be doing dropna here. This will make a block of dates with even 1 missing value discontinuous which I think is less beneficial than using it and filling in the missing timestamp later on (e g one missing point can cost you something like 45 potential t0s with 3h history and 8h forecast). I've used data with about 3% missing, sometimes in considerable chunks, and the model seemed to do fine and not get distracted by nan infills.

There is a greater discussion to be had around how much missing data we allow to be infilled and at what times, but I think this should be done in preprocessing anyway and not here; I'd remove it for now.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The relevant entry point is ocf_data_sampler/torch_datasets/site.py, around the site loop and site.dropna(dim='time_utc'); read how valid_t0_and_site_ids is built after this point. Confirm the change preserves timestamps across missing values and avoids discarding potential t0s, then check the resulting sampling behavior against the issue's 3-hour history and 8-hour forecast example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.