arviz-devs / arviz-devs/InferenceObjects.jl
Reducing compilation time with InferenceData
- Dominant language
- Julia
- Stars
- 15
- Forks
- 1
- Avg merge
- 6d 18h
- Merged PRs (30d)
- 1
Description
With DimensionalData everything is strongly typed, and we've adopted that for `Dataset` and `InferenceData` as well, where the underlying storage is a `NamedTuple`. One of the consequences of this is that if a user adds a variable to a group, then the resulting `InferenceData` now has a new type, and so there are frequent delays due to JIT compiling.
It would be nice if we could figure out a workaround for this. DataFrames seems to do so by having the underling storage be an `OrderedCollections.LittleDict`. I'm guessing DataFrames maintains efficiency when operating on columns/rows by using function barriers everywhere. We could do something similar. This would also allow `InferenceData` and `Dataset` to be modified in-place. Type inferrability for efficiency is likely only critical when operating on variables themselves.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading how Dataset and InferenceData use NamedTuple storage and how variables are added to groups. Compare the proposed OrderedCollections.LittleDict and function-barrier approach with the DataFrames design. Done means reducing JIT compilation delays while preserving efficient variable operations and enabling in-place modification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100