JuliaParallel / JuliaParallel/DistributedArrays.jl

Unnecessary memory allocations in restrict_indices

Open
#215 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
205
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Hello,

I found this when I was puzzled by a [discourse question](https://discourse.julialang.org/t/strange-memory-allocation-with-distributedarrays/38325).

The function `restrict_indices` (called on line 824 in darray.jl, for example, couldn't figure out how to reference here), allocates an array of indices as big as the array that that you are trying to copy. Thus doubling the memory allocations.

Not sure if there is a way around it (just allocating tuples of ranges of indices or the like), but I thought I should file an issue so it doesn't get lost.

Cheers!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading restrict_indices and its call at line 824 of darray.jl, then inspect how the index array is created during copying. Confirm the reported allocation behavior from the linked Discourse question. Done means avoiding the unnecessary array-sized allocation while preserving restrict_indices behavior and distributed array copying.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.