JuliaParallel / JuliaParallel/DistributedArrays.jl

Write DArray to HDF5/JLD without first converting to Array

オープン
#108 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Julia
スター
205
フォーク
34
PR マージ指標
30日以内にマージされた PR はありません

説明

It would be useful to have a way to write `DArray`s to file without first collecting all of the data on the local process. I found a pretty hacky way of doing this that looks like this:

``` julia
function write_darray{T<:AbstractFloat}(filepath::AbstractString, darr::DArray{T})
function write_localpart(pid::Int)
jldopen(filepath, "r+") do file
write(file, "inds$pid", collect(localindexes(darr)))
write(file, "arr$pid", localpart(darr))
end
end

# Write DArray metadata
jldopen(filepath, "w") do file
write(file, "dims", darr.dims)
write(file, "pids", collect(darr.pids))
end

# For each process, write the local indices and local part
for pid in darr.pids
remotecall_wait(pid, write_localpart, pid)
sleep(0.001) # This is for some reason necessary, or else HDF5 complains that the new object already exists
end
end

function read_darray(filepath::AbstractString)
file = jldopen(filepath, "r")
dims = read(file, "dims")
pids = read(file, "pids")
out = zeros(dims...)

# Reassemble local parts
for pid in pids
inds = read(file, "inds$pid")
out[inds...] = read(file, "arr$pid")
end

close(file)
return out
end
```

However, there must be a better way than this, especially one that would create a single variable in the file `filepath` (rather than `2n+2` in my case, where `n` is the number of processes the `DArray` is stored on).

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Start with the issue's write_darray and read_darray examples, including jldopen, localindexes, localpart, and remotecall_wait. Investigate how DistributedArrays and the HDF5/JLD layer could represent one distributed variable rather than per-process objects. Done means a supported approach can write and read a DArray without collecting it locally and without creating 2n+2 file variables.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
julia
領域
distributed-systems
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。