NatLabRockies / NatLabRockies/GridAnalysisToolkit
Slow / apparently-hung first import from a fresh pip install
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Reported: a user installed nlr-gat from PyPI into a brand-new venv and ran from gat.scenariohandlers import PlexosScenario in the REPL. It appeared to hang — no output for a long time — and they Ctrl-C'd it. (The retry then hit pandas's known circular-import breakage after an interrupted import, which is a separate, expected consequence of the Ctrl-C, not a GAT bug.) Given time, the import does complete.
Reproduction and measurement
Fresh uv venv, pip install nlr-gat (built from current main), then:
$ time python -c "from gat.scenariohandlers import PlexosScenario"
...
2.20s user 0.58s system 5% cpu **51.216 total**
~51s wall clock but only ~2.8s of CPU time — the process is blocked, not computing. Re-running the identical command immediately after:
$ python -X importtime -c "from gat.scenariohandlers import PlexosScenario"
...
gat.scenariohandlers cumulative: 1,365,155 µs (~1.37s)
So there are two distinct, additive problems:
1. One-time macOS Gatekeeper/codesign tax on first load (environmental, not a GAT bug per se)
The 51s-vs-1.37s gap matches the well-known macOS behavior where Gatekeeper/XProtect scans every newly-extracted compiled binary (.so/.dylib) the first time it's dlopen'd, before trusting it. GAT's dependency closure includes several large native-code packages — geopandas' GDAL/pyproj/shapely bindings, DuckDB, PyArrow, and (on macOS, via gat.quickplots's backend-selection logic) PySide6/Qt, which alone ships hundreds of dylibs. That's a lot of first-touch scanning on a fresh install.
We can't eliminate the OS-level scan, but we should stop making it worse than it needs to be (see #2) and set expectations: document in the README/troubleshooting that the very first import after installing can take noticeably longer on macOS, and it is not a hang.
2. Real, avoidable eager-import overhead (~1s, the addressable part)
Even on a warm cache, importing PlexosScenario alone pulls in far more than PLEXOS-loading needs, because gat/scenariohandlers/__init__.py eagerly imports every handler:
from .sienna import SiennaScenario
from .plexos import PlexosScenario
from .egret import EGRETScenario
from .reeds import ReEDsScenario
from .multi import MultiScenario
from .base import BaseScenario
from .file_scenario import FileScenario
This is inconsistent with the lazy __getattr__ pattern already used in the top-level gat/__init__.py (see its load/load_scenario_only/etc. lazy imports) — scenariohandlers never got the same treatment.
Consequences, from the -X importtime breakdown (cumulative µs):
| Module | Cumulative | Why it's pulled in for a PLEXOS-only import |
|---|---|---|
gat.quickplots / .utils |
337,949 / 337,966 | plexos.py imports random_color from gat.quickplots.utils — drags in matplotlib, matplotlib.pyplot, and macOS Qt-backend probing (matplotlib.backends.qt_compat, 81,706 µs) merely for one color-picking helper |
gat.datahelpers (→ .sienna, geopandas) |
258,236 / 139,204 / 133,207 | scenariohandlers/base.py does from gat.datahelpers.parsers import *, which runs gat/datahelpers/__init__.py's from .sienna import * — pulling in geopandas (GDAL/pyproj/shapely) for a handler that never touches Sienna or geo data |
gat.simulations (→ duckdb, polars) |
274,855 / 101,838 / 163,059 | pulled in via the eager SiennaScenario import even when only PlexosScenario was requested |
gat.datahelpers.h5Parsers / h5py |
114,691 / 107,424 | same eager-datahelpers-package cause |
Baseline pandas (necessary regardless) is 377,657 µs — so roughly ~1s of the ~1.37s total is avoidable overhead from importing code paths the caller didn't ask for.
Suggested fixes
- Make
gat/scenariohandlers/__init__.pylazy, mirroringgat/__init__.py's__getattr__pattern, sofrom gat.scenariohandlers import PlexosScenarioonly importsplexos.pyand its actual transitive dependencies. - Stop
plexos.py(and any other handler) from importinggat.quickplotsat module load just forrandom_color()— either inline a tiny color helper with no matplotlib dependency, or import it lazily inside the function that uses it. This also means fewer PLEXOS-only users pay the Qt-backend-detection cost at all. - Don't blanket-
import *geopandas-dependent submodules fromgat/datahelpers/__init__.py—sienna.py's geopandas usage should be lazy/scoped so importinggat.datahelpers.parsers(whichbase.pyneeds) doesn't transitively import geopandas for handlers that never touch it. - Consider the same lazy treatment for
gat.simulations' duckdb/polars-backed submodules.
3. Related: unsolicited DEBUG log spam on plain import
Because @plot_function-decorated functions register themselves at module import time (in gat/quickplots/{dispatch,transmission,multi_system}.py), and GAT never calls logger.remove() outside the CLI's setup_cli_logging(), a plain import gat / from gat.scenariohandlers import ... in a fresh interpreter prints loguru's default DEBUG-level messages straight to stderr, e.g.:
2026-08-01 ... | DEBUG | gat.registry:decorator:68 - Registered plot function: plot_generation_capacities for MultiScenario
For a Python-API user (as opposed to gat CLI users, who get setup_cli_logging), this looks like noise or an error at best. As a library, GAT shouldn't leave a DEBUG-level default sink active for API consumers — either don't rely on loguru's bundled default handler (call logger.remove() at package import, matching common library practice of staying silent until the app configures logging) or defer plot-function registration out of import time entirely (fits naturally with fix #1/#2 above, since import-time registration is exactly what an __getattr__-based lazy scenariohandlers would also need to stop doing implicitly).
None of this is fatal — the import does complete — but combined, a first-time PyPI user's very first import can look exactly like a hang, and the underlying eager-import chain is worth trimming regardless of the OS-level tax.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with gat/scenariohandlers/init.py and compare its eager imports with the lazy patterns in gat/init.py. Trace the imports from gat/scenariohandlers/plexos.py, gat/quickplots.utils, and gat/datahelpers/init.py, then rerun the documented warm and fresh -X importtime commands. Done means the PLEXOS import avoids unrelated handlers and dependencies, completes faster, and does not emit unsolicited DEBUG logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- matplotlib, pandas, python
- Domain
- developer-experience, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100