python / python/mypy

Fine-grained incremental step does lots of spurious stat() calls

Offen
#5,747 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

priority-1-normal topic-daemon topic-fine-grained-incremental
Vorherrschende Sprache
Python
Sterne
20.6k
Forks
3.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

While working on #5745 I instrumented mypy/fscache.py and discovered that a lot of spurious os.stat() calls are made in fine-grained-incremental runs on behalf of find_module() (in mypy/modulefinder.py). This is invoked from is_module() (in mypy/build.py) whose only call site is in all_imported_modules_in_file(). This is used by compute_dependencies() to disambiguate from X import Y -- it needs to know whether Y is a submodule of X or just some object (like a function or class) defined in X. This in turn happens during an incremental call to load_graph() made from update_module_isolated() (in mypy/server/update.py), which is intended to recreate the mypy.build.State object for a specific module that is known to be changed in a run.

For example, the line from typing import Dict ends up calling stat() for the following files in my setup:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/typing-stubs
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/typing/py.typed
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/typing/Dict/py.typed
/Library/Python/3.6/site-packages/typing-stubs
/Library/Python/3.6/site-packages/typing/py.typed
/Library/Python/3.6/site-packages/typing/Dict/py.typed
/Users/guido/Library/Python/3.6/lib/python/site-packages/typing-stubs
/Users/guido/Library/Python/3.6/lib/python/site-packages/typing/py.typed
/Users/guido/Library/Python/3.6/lib/python/site-packages/typing/Dict/py.typed
typing
/Users/guido/src/mypy/typing
/Users/guido/src/mypy/mypy/typeshed/stdlib/3.6/typing
/Users/guido/src/mypy/mypy/typeshed/stdlib/3.5/typing
/Users/guido/src/mypy/mypy/typeshed/third_party/3.5/typing
/Users/guido/src/mypy/mypy/typeshed/stdlib/3/typing
/Users/guido/src/mypy/mypy/typeshed/third_party/3/typing
/Users/guido/src/mypy/mypy/typeshed/stdlib/2and3/typing
/Users/guido/src/mypy/mypy/typeshed/third_party/2and3/typing
/usr/local/lib/mypy/typing

Note that all these stat() calls are cached by fscache.py, but the cache is flushed at the end of each incremental step, so each incremental step does each of these once. (Also note that the current directory seems to appear twice on the search path, once as '', once as its absolute path.)

I came up with a tentative fix, but it needs work (to account for typeshed) and since this is pre-existing behavior I decided to separate it from #5745.

It's not the end of the world, obviously, but I think this ends up doing ~20 stat() calls for each from import (though only in modified files), and since we're looking for a strategy to allow using Watchman instead of calling stat() for each source file, I think we might want to do something about this. (I also happen to know that from import is hugely popular in the Dropbox code base.) I'll assume this is low priority until I have determined how many stat() calls this does for the typical real Dropbox use case.

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Verfolge den inkrementellen Pfad von mypy/server/update.py:update_module_isolated() durch mypy/build.py, mypy/modulefinder.py und mypy/fscache.py, mit Fokus auf all_imported_modules_in_file() und compute_dependencies(). Vergleiche das feingranulare inkrementelle Verhalten mit den typeshed-Suchpfaden und dem vorläufigen Fix in PR 5745; als erledigt gilt, wenn unnötige stat()-Aufrufe vermieden werden und zugleich die korrekte Unterscheidung zwischen Submodul und Objekt erhalten bleibt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
compilers, performance
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
38/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.