JuliaParallel / JuliaParallel/Dagger.jl

Detect network structure for improved scheduling and parallelism

Open
#126 0 comments 0 reactions 0 assignees View on GitHub
enhancement file io performance scheduler
Dominant language
Julia
Stars
723
Forks
90
Avg merge
1d 37m
Merged PRs (30d)
9

Description

Modern clusters (especially supercomputers) usually have more than one networking device between a given pair of nodes, and specialized network fabrics (e.g. Infiniband, NVLINK, ROCmRDMA, etc.) which can provide optimized transfers between processors when utilized. Our current processor infrastructure only represents (combined) memory and compute domains, but does not model the connections between and within domains. If we were able to query the network/memory connections between processors, and then transfer data between processors using only a specifically-selected connection, we could enable the scheduler to make much more efficient decisions about where to place work to optimize data transfer latencies and decrease network bottlenecking.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.