NCAR / NCAR/DART

feat req: Scalable obs sequences

Open
#745 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Fortran
Stars
263
Forks
182
Avg merge
11d 12h
Merged PRs (30d)
7

Description

Currently every processor in DART reads the entire observation sequence into memory.

total memory = obs_seq_size * num_procs

Fig from Kamil Yousuf:
Screenshot 2024-09-30 at 10 34 14 AM

In addition, the obs sequence reads and writes are single processor, which anti-scales.

This is no longer sufficient

  • increasing number of observations (e.g. satellite obs)
  • high resolution DA (large core count, wasting cycles with singe core IO)
  • particularly when obs sequence contain external forward operators (more per core memory). Side note: the obs sequence is maybe not the place to read/write external FOs, but that is the current design.
  • AI models (may want these to be subroutine callable and run many windows in one filter run)

Kamil Yousuf, Rhodes College SiParCS worked on reading obs sequences for multiple time windows: ~1/2 billion observations read and distributed. Kamil also has as parallel sort, and is working on parallel writes.
Kamil is assuming that the observation length is calculable (calculatable?, predictable), which is not guaranteed in general currently (but can be).
Kamil's fork https://github.com/tyiop794/DART (also has obs seq test harness)

Folder in Specs for Obs_Seq_IO

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Obs_Seq_IO Specs folder and Kamil Yousuf's DART fork, including its observation-sequence test harness. Compare the existing single-processor reads and writes with the fork's work on reading and distributing multiple time windows, parallel sorting, and parallel writes. Done should address scalable observation-sequence I/O without every processor loading the entire sequence, including the observation-length concern.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
data-engineering, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.