biocore / biocore/BIRDMAn

concatenate_inferences is slow

Open
#57 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
25
Forks
6
PR merge metrics
No merged PRs in 30d

Description

I'm noticing that sometimes the [concatenate_inferences](https://github.com/gibsramen/BIRDMAn/blob/main/birdman/model_util.py#L117) method is the slowest part of the computation, even slower than MCMC sampling.

It looks like this can be speed up with dask -- the trick is to rechunk your `az.InferenceData` object, and I believe dask will do the rest (so no need to implement here I think). It does become very problematic when concatenating `az.InferenceData` objects with tens of thousands of features; possibly because the dask scheduler gets overwhelmed and all operations become single-threaded. I've raised this issue on the [xarray discussions](https://github.com/pydata/xarray/discussions/5657)

In which case, the workaround is to turn `concatenate_inferences` into a reduction operation (i.e. merge only 1000 datasets at a time, and then merge those together).

Mainly raising this as an issue because this method is going to be problematic for larger datasets, and a reduce version of this function maybe necessarily.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at birdman/model_util.py#L117 and profile concatenate_inferences with az.InferenceData containing tens of thousands of features. Read the linked xarray discussion and investigate rechunking and the proposed reduction approach of merging batches of 1000 datasets. Done means concatenation no longer becomes the slowest part of computation on large datasets.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.