feat req: Scalable obs sequences
Nobody has claimed this yet.
- Dominant language
- Fortran
- Stars
- 263
- Forks
- 182
- Avg merge
- 11d 12h
- Merged PRs (30d)
- 7
Description
Currently every processor in DART reads the entire observation sequence into memory.
total memory = obs_seq_size * num_procs
Fig from Kamil Yousuf:
In addition, the obs sequence reads and writes are single processor, which anti-scales.
This is no longer sufficient
- increasing number of observations (e.g. satellite obs)
- high resolution DA (large core count, wasting cycles with singe core IO)
- particularly when obs sequence contain external forward operators (more per core memory). Side note: the obs sequence is maybe not the place to read/write external FOs, but that is the current design.
- AI models (may want these to be subroutine callable and run many windows in one filter run)
Kamil Yousuf, Rhodes College SiParCS worked on reading obs sequences for multiple time windows: ~1/2 billion observations read and distributed. Kamil also has as parallel sort, and is working on parallel writes.
Kamil is assuming that the observation length is calculable (calculatable?, predictable), which is not guaranteed in general currently (but can be).
Kamil's fork https://github.com/tyiop794/DART (also has obs seq test harness)
Folder in Specs for Obs_Seq_IO
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Obs_Seq_IO Specs folder and Kamil Yousuf's DART fork, including its observation-sequence test harness. Compare the existing single-processor reads and writes with the fork's work on reading and distributing multiple time windows, parallel sorting, and parallel writes. Done should address scalable observation-sequence I/O without every processor loading the entire sequence, including the observation-length concern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100