JuliaArrays / JuliaArrays/ArrayInterface.jl

Managing array memory

Open
#104 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
140
Forks
44
Avg merge
4d 14h
Merged PRs (30d)
5

Description

The formal interface for working with an array's memory is currently very limited. We mostly just have ways of accessing preallocated memory (e.g., unsafe_wrap and unsafe_pointer_to_objref). The rest of the time we need to rely on methods like push!/pop!/append!/resize to mutate a Vector down the line. Likewise, if we want to allocate memory for a new array we typically need to wrap a new instance of Array.

A couple reasons we this would be good to have here

  1. Performance. The necessity to ensure that memory is safely managed also means that each call to something like resize! has to ensure that we don't create a situation where we are out of bounds. This probably only has minimal overhead in most cases but if we are doing something complicated like merging and or sorting two vectors then we may be calling resize! a lot. I recall this sort of thing coming up a lot when I was trying to work with graph algorithms last year that do a lot of insertion and deletion.
  2. Necessary for creating some interfaces. One of the things that makes the indexing interface so nice is that there are formally defined places for new types to insert and propagate information (e.g., bounds checking). It also makes it easier to optimize indexing. In contrast, Vector directly uses methods that allocate and write to memory (e.g., Base._growend! and Base.arrayset!) while abstract types typically hope that resize! works and then use setindex!.

From where I'm standing it seams like this would be great to have here. The reason this is an issue instead of a PR is mainly because I'm unsure if the implementation for this would be too general or involved for this package. We could always define several methods here like unsafe_grow_end! and unsafe_shrink_end! but these aren't super helpful without implementations that can interact with pointers and references.

@chriselrod, do you think growing/shrinking/allocating memory in an efficient way would require a bunch of LLVM magic or could we do it pretty simply here?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing unsafe_wrap, unsafe_pointer_to_objref, Base._growend!, and Base.arrayset! interfaces and how resize! is used for vectors. Determine whether efficient growing, shrinking, and allocation can be exposed with a sufficiently general interface, then document a concrete scope and implementation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.