concatenate_inferences is slow
- Dominant language
- Python
- Stars
- 25
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
I'm noticing that sometimes the [concatenate_inferences](https://github.com/gibsramen/BIRDMAn/blob/main/birdman/model_util.py#L117) method is the slowest part of the computation, even slower than MCMC sampling.
It looks like this can be speed up with dask -- the trick is to rechunk your `az.InferenceData` object, and I believe dask will do the rest (so no need to implement here I think). It does become very problematic when concatenating `az.InferenceData` objects with tens of thousands of features; possibly because the dask scheduler gets overwhelmed and all operations become single-threaded. I've raised this issue on the [xarray discussions](https://github.com/pydata/xarray/discussions/5657)
In which case, the workaround is to turn `concatenate_inferences` into a reduction operation (i.e. merge only 1000 datasets at a time, and then merge those together).
Mainly raising this as an issue because this method is going to be problematic for larger datasets, and a reduce version of this function maybe necessarily.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at birdman/model_util.py#L117 and profile concatenate_inferences with az.InferenceData containing tens of thousands of features. Read the linked xarray discussion and investigate rechunking and the proposed reduction approach of merging batches of 1000 datasets. Done means concatenation no longer becomes the slowest part of computation on large datasets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100