astropy / astropy/specutils

slow import times, partially due to astropy.nddata and dask

Open
#1,174 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
204
Forks
135
Avg merge
12h 55m
Merged PRs (30d)
1

Description

We were recently studying slow import times in DESI code (desihub/desispec#2379) which was partially traced to specutils, which has a chain of imports that include things like dask and matplotlib.

This will print a dizzying amount of information about what is imported and what takes time
```
python -Ximporttime -c "import specutils"
```
selected lines:
```
import time: self [us] | cumulative | imported package
import time: 2351 | 1548033 | dask.array
import time: 2059 | 2833833 | astropy.nddata.nddata
import time: 1379 | 2835211 | astropy.nddata.decorators
import time: 1200 | 2836411 | astropy.nddata.blocks
import time: 2140 | 2853724 | astropy.nddata
import time: 4477 | 4477 | mpl_toolkits.axes_grid1.anchored_artists
import time: 1667 | 6144 | astropy.visualization.wcsaxes.helpers
import time: 3214 | 3214 | astropy.visualization.wcsaxes.patches
import time: 186643 | 228080 | astropy.visualization.wcsaxes
import time: 1561 | 229640 | ndcube.visualization.mpl_plotter
import time: 1950 | 794716 | ndcube.ndcube
import time: 3336 | 826281 | ndcube
import time: 1658 | 4635222 | specutils.spectra.spectrum1d
import time: 2445 | 4640044 | specutils.spectra
import time: 57960 | 5366278 | specutils
```

i.e. the overall specutils import took 5.3 seconds, of which 2.8 were from astropy.ndata, of which 1.5 were from dask.array even though we don't use any dask at all, we just happen to also have it installed. The matplotlib impact was smaller, but still surprising to see.

This smells like an underlying astropy issue, but I'm starting with filing it here because I was just chatting with @tepickering who suggested I post here, and then specutils might sort this out with astropy.

Our developement versions are a little on the old side, so it would be worth having a specutils developer retest with latest versions before diving into this too much more.

python 3.10.14
specutils 1.14.0
astropy 6.0.1

Contributor guide

Open the contributing guide

Research direction

Start by rerunning `python -Ximporttime -c "import specutils"` with current versions, then inspect the reported import chain through `specutils`, `astropy.nddata`, `ndcube`, dask, and matplotlib. Compare the timings and determine whether the unnecessary imports can be addressed in specutils or should be reported upstream to Astropy; done means documenting a reproducible finding and an agreed location for the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.