JuliaClimate / JuliaClimate/meta
Example notebooks exhibiting the JuliaClimate stack
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 7
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
It would be nice to have some examples of the existing JuliaClimate stack in action in notebooks (if there redundancies, then maybe even feature / performance comparisons?).
My personal goal for the next few weeks is to basically implement a Julia translation of my "big data" Python tutorial [binder link] which uses the following stack:
xarrayanddaskas the building blocks for reading and analyzing out-of-memory labelled NetCDF arraysintake(generic organizational tool leveraging xarray) to read in netcdf files stored in a cloud-optimized zarr format in Google Cloud storage as xarray.Dataset instancesxgcmfor doing grid-aware operations (e.g. differentiation) on datasetsxmitgcm(model-specific package) to process the dataset's non-rectangular native grid into something more rectangular
These are basically the four categories of packages that I see as necessary to replicate the kinds of workflows that I am interested (and which are extremely straight-forward using the existing Pangeo Python-stack, as in my example above):
- Basic data types that make handling large NetCDF-like datasets efficient (low overhead), effortless (intuitive and compact syntax), scalable (distributable, out-of-memory), and extendable (flexible and simple data structure types).
- Organizational packages that simplify the workflow (e.g. organizing model ensembles, models vs. observations, downloading scripts)
- Generic utility packages that extend the functionality of 1-type packages (e.g. for problem-specific, dimension-specific, or operation-specific uses).
- Model-specific utility packages that rely on model-specific metadata (e.g. some of what @natgeo-wong and @gaelforget are working on).
I don't really know how much sense it makes in putting effort to develop 2, 3, and 4 if we haven't yet settled on a stable 1.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with notebooks/Lecture04_dask_and_big_data-serial.ipynb and its linked Binder example to understand the proposed workflow and compare it with the JuliaClimate packages named in the issue. Done would mean a defined set of working notebooks that demonstrates the stack and clarifies which package categories are covered, without leaving the scope dependent on unresolved stack choices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, julia, jupyter-notebook, python
- Domain
- data-engineering, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100