google / google/sbsim

Weather Data Cleaning

Open
#144 2 comments 0 reactions 1 assignee Claimed by @s2t2 View on GitHub
internal
Dominant language
Python
Stars
123
Forks
56
Avg merge
17d 18h
Merged PRs (30d)
3

Description

> NOTE: this issue references a future state of the code that has not yet been synced out to github

We see the weather data uses extreme values like -9999.0 to signal a null or bad value. For example in the "TempC" column.

To avoid downstream issues when using this data in simulation, we should model those values as null or nan instead.

We should then also make sure the Replay Weather Controller appropriately handles / drops those null values (instead of using a very extreme negative number). Using the interpolation strategy, the absence of one hour won't negatively impact the results because the weather controller will use the average between the closest two hours it does have.

For the "TempC" column, the following dates have a single instance of these bad / extreme values:

2025:
+ 04-02
+ 06-06
+ 07-07
+ and possibly others not included since the current data snapshot ends before the end of the year

2024:

+ 06-04
+ 07-04
+ 08-03
+ 11-16
+ 11-24
+ 12-24

2023:

+ 04-20

As a temporary workaround we are performing the filtering downstream:

```py
wc = ReplayWeatherController()
# or access from the environment:
# wc = env.building.simulator.weather_controller
wc.weather_df = wc.weather_df[ wc.weather_df["TempC"] != -9999.0]
# now you can use the weather controller
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.