JuliaClimate / JuliaClimate/ClimateBase.jl

[FR] Out-of-memory data reduction.

Open
#48 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

easy feature request hard
Dominant language
Julia
Stars
38
Forks
3
PR merge metrics
No merged PRs in 30d

Description

While the in-memory functionality is great, it is typically the case that you have so much data that they don't fit to memory. Typically these data are saved in either monthly or yearly files, where each file contains one year of all the data, etc.

This is good for us, because at the moment it isn't hard to write a simple for-loop over your code. However we can streamline many things. For example, the output ClimArray can be pre-initialized and efficiently aggregated over, similarly to how yearlyagg works now.

So in principle there are two ways to do out-of-memory data reduction:

  1. Reduce by aggregating over time, by reducing the total amount of time-points and doing an out-of-memory version of yearlyagg and looping over the files.
  2. Reduce by projecting to a lower resolution grid. This is done for each time slice in the files, once again looping over files. This will require us to have #46 ready so that we can use it here.

The above is in my eyes easy, provided that the required issues are solved first.

The thing that is hard is also getting automatic parallelization to work here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing yearlyagg behavior and the work tracked in issue #46. Define how file-by-file temporal aggregation and lower-resolution projection should pre-initialize and update a ClimArray, then determine how automatic parallelization fits both approaches. Done means out-of-memory reduction works across the described file layouts and its parallel behavior is specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.