JuliaData / JuliaData/PooledArrays.jl
Redesing of PooledArray internals
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 49
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
Things to do:
- treat
missingas a special value that is not pooled, probably with level0. This would work the same as in CategoricalArrays.jl; the benefit is that twoPooledArraysdiffering only in the fact if they allowMissingor not could share pool - add locking for
setindex!but make sure that we support batch operations of adding levels (both insetindex!and in e.g.copyto!); this will allow to fully drop Copy-On-Write and never copy pool and invpool by default; tentativelyunsafe_setindex!would be an alternative that does not use lock - stress in documentation that using
invpoolis not safe if potentially other threads are modifying it (this should not be a problem) - add
droplevels!to DataAPI.jl and to PooledArrays.jl (this requires also a change in CategoricalArrays.jl); this function would reduce pool and invpool to only used levels and also at the same time make a fresh copy of them (as a way to detach pool and invpool between PooledArray-s)
I think this design is better than global pool. It will still cost us a bit in H2O benchmarks, but at least we avoid a global pool that is not reclaimable.
@nalimilan + @quinnj : any additional comments on this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the current PooledArrays.jl pool and invpool internals, then compare the related behavior in CategoricalArrays.jl. Check the DataAPI.jl integration before deciding how missing values, locking, droplevels!, and documentation should fit together; the work is done when all four design goals are implemented consistently across the affected packages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100