pytroll / pytroll/satpy

Importing satpy getting slower

Open
#2,159 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

optimization
Dominant language
Python
Stars
1.2k
Forks
335
Avg merge
1d 18h
Merged PRs (30d)
10

Description

As satpy becomes more complex, it is importing more and more libraries. This is becoming problematic for us as we're often running tens of thousands of very small python jobs that don't need many of satpy's features.
For example, according to python -X importtime -c "import satpy" it takes ~2.3 seconds to import the most recent version of satpy, of which ~0.3 seconds is importing rasterio. However, many of our scripts make no use of rasterio, so this import is unneeded.

Is there any way to alter the code structure so that imports are only made when needed? Could, for example, the rasterio imports be moved into the writer so they're only imported when save_dataset is called?

For reference, I ran some stats of how long importing satpy takes across three versions (the older two chosen at random, tbh):
v0.18: 1.95 seconds
v0.25: 2.18 seconds
v0.36: 2.41 seconds

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with python -X importtime -c "import satpy" and trace the imports responsible for the reported rasterio cost. Inspect the writer path around save_dataset to identify imports that can be deferred; done means unused features no longer load those libraries while the relevant save behavior still works and import timing improves.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.