JuliaClimate / JuliaClimate/meta

Example notebooks exhibiting the JuliaClimate stack

Open
#4 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
7
Forks
1
PR merge metrics
No merged PRs in 30d

Description

It would be nice to have some examples of the existing JuliaClimate stack in action in notebooks (if there redundancies, then maybe even feature / performance comparisons?).

My personal goal for the next few weeks is to basically implement a Julia translation of my "big data" Python tutorial [binder link] which uses the following stack:

  1. xarray and dask as the building blocks for reading and analyzing out-of-memory labelled NetCDF arrays
  2. intake (generic organizational tool leveraging xarray) to read in netcdf files stored in a cloud-optimized zarr format in Google Cloud storage as xarray.Dataset instances
  3. xgcm for doing grid-aware operations (e.g. differentiation) on datasets
  4. xmitgcm (model-specific package) to process the dataset's non-rectangular native grid into something more rectangular

These are basically the four categories of packages that I see as necessary to replicate the kinds of workflows that I am interested (and which are extremely straight-forward using the existing Pangeo Python-stack, as in my example above):

  1. Basic data types that make handling large NetCDF-like datasets efficient (low overhead), effortless (intuitive and compact syntax), scalable (distributable, out-of-memory), and extendable (flexible and simple data structure types).
  2. Organizational packages that simplify the workflow (e.g. organizing model ensembles, models vs. observations, downloading scripts)
  3. Generic utility packages that extend the functionality of 1-type packages (e.g. for problem-specific, dimension-specific, or operation-specific uses).
  4. Model-specific utility packages that rely on model-specific metadata (e.g. some of what @natgeo-wong and @gaelforget are working on).

I don't really know how much sense it makes in putting effort to develop 2, 3, and 4 if we haven't yet settled on a stable 1.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with notebooks/Lecture04_dask_and_big_data-serial.ipynb and its linked Binder example to understand the proposed workflow and compare it with the JuliaClimate packages named in the issue. Done would mean a defined set of working notebooks that demonstrates the stack and clarifies which package categories are covered, without leaving the scope dependent on unresolved stack choices.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, julia, jupyter-notebook, python
Domain
data-engineering, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.