arviz-devs / arviz-devs/InferenceObjects.jl

Reducing compilation time with InferenceData

Open
#15 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Julia
Stars
15
Forks
1
Avg merge
6d 18h
Merged PRs (30d)
1

Description

With DimensionalData everything is strongly typed, and we've adopted that for `Dataset` and `InferenceData` as well, where the underlying storage is a `NamedTuple`. One of the consequences of this is that if a user adds a variable to a group, then the resulting `InferenceData` now has a new type, and so there are frequent delays due to JIT compiling.

It would be nice if we could figure out a workaround for this. DataFrames seems to do so by having the underling storage be an `OrderedCollections.LittleDict`. I'm guessing DataFrames maintains efficiency when operating on columns/rows by using function barriers everywhere. We could do something similar. This would also allow `InferenceData` and `Dataset` to be modified in-place. Type inferrability for efficiency is likely only critical when operating on variables themselves.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading how Dataset and InferenceData use NamedTuple storage and how variables are added to groups. Compare the proposed OrderedCollections.LittleDict and function-barrier approach with the DataFrames design. Done means reducing JIT compilation delays while preserving efficient variable operations and enabling in-place modification.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.