AusClimateService / AusClimateService/plotting_maps

Apply Dask to parallelise acs_regional_stats

Offen
#34 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Jupyter Notebook
Sterne
8
Forks
1
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

`acs_regional_stats` can be very memory intensive to run, particularly over many regions and many timesteps.
We should develop an example of running `acs_regional_stats` for many years of daily data to produce area averaged timeseries for regions. Currently, this is [possible](https://github.com/AusClimateService/plotting_maps/blob/main/example_notebooks/FAQ_example_timeseries_stats.ipynb), but will take several minutes to calculate.
Dask is likely to be able to achieve this by calculating area averages per file.
Previous development has focused on reducing memory usage through other clever means, such as implementing `chunks` to reduce the number of timesteps loaded into the memory to calculate stats over each time. This could be parallelised, but it is not currently.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with example_notebooks/FAQ_example_timeseries_stats.ipynb and the acs_regional_stats entry point, then run the current multi-year daily calculation to establish its memory use and runtime. Develop the requested Dask-based example that calculates area averages per file, and verify that it produces regional time series while reducing the current several-minute, memory-intensive workload.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
jupyter-notebook
Bereich
data, performance
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.