Importing satpy getting slower
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 335
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 10
Description
As satpy becomes more complex, it is importing more and more libraries. This is becoming problematic for us as we're often running tens of thousands of very small python jobs that don't need many of satpy's features.
For example, according to python -X importtime -c "import satpy" it takes ~2.3 seconds to import the most recent version of satpy, of which ~0.3 seconds is importing rasterio. However, many of our scripts make no use of rasterio, so this import is unneeded.
Is there any way to alter the code structure so that imports are only made when needed? Could, for example, the rasterio imports be moved into the writer so they're only imported when save_dataset is called?
For reference, I ran some stats of how long importing satpy takes across three versions (the older two chosen at random, tbh):
v0.18: 1.95 seconds
v0.25: 2.18 seconds
v0.36: 2.41 seconds
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with python -X importtime -c "import satpy" and trace the imports responsible for the reported rasterio cost. Inspect the writer path around save_dataset to identify imports that can be deferred; done means unused features no longer load those libraries while the relevant save behavior still works and import timing improves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 40/100