JuliaClimate / JuliaClimate/ClimateBase.jl
[FR] Out-of-memory data reduction.
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 38
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
While the in-memory functionality is great, it is typically the case that you have so much data that they don't fit to memory. Typically these data are saved in either monthly or yearly files, where each file contains one year of all the data, etc.
This is good for us, because at the moment it isn't hard to write a simple for-loop over your code. However we can streamline many things. For example, the output ClimArray can be pre-initialized and efficiently aggregated over, similarly to how yearlyagg works now.
So in principle there are two ways to do out-of-memory data reduction:
- Reduce by aggregating over time, by reducing the total amount of time-points and doing an out-of-memory version of
yearlyaggand looping over the files. - Reduce by projecting to a lower resolution grid. This is done for each time slice in the files, once again looping over files. This will require us to have #46 ready so that we can use it here.
The above is in my eyes easy, provided that the required issues are solved first.
The thing that is hard is also getting automatic parallelization to work here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing yearlyagg behavior and the work tracked in issue #46. Define how file-by-file temporal aggregation and lower-resolution projection should pre-initialize and update a ClimArray, then determine how automatic parallelization fits both approaches. Done means out-of-memory reduction works across the described file layouts and its parallel behavior is specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100