python / python/cpython

Uses a higher granularity than seconds for `.pyc` file invalidation

Offen
#121,376 19 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

type-feature
Vorherrschende Sprache
Python
Sterne
77.2k
Forks
35.9k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Bug report

Bug description:

reproduction:

#!/bin/sh 
 
mkdir -p src 
rm -fr src/__pycache__ 
rm -fr __pycache__ 
 
echo 'from src.test import f' > main.py 
echo 'print(f())' >> main.py 

echo 'def f():' > src/test.py 
echo '    return "Hello!"' >> src/test.py 
# prints "Hello!"
python3 main.py 

echo 'def f():' > src/test.py 
echo '    return "Goodbye!"' >> src/test.py 
# also prints "Hello!", but note the file does not say that anymore
python3 main.py

(this is probably sensitive to how fast your computer is)

this seems to stem from (i guess) that the timestamps are 32-bit, and write the current second (unix time, as 32-bit). and then, when the file is edited and the python script runs again, it gets a cache hit (same timestamp as file 32-bit mtime) and yields the old content.

is this known/intended? is there a mitigation that isn't just "delete the pycache"? i tried looking if anyone else ran into this and could not find anything.

i ran into this as part of a workflow that first edited a python script via shell (edits mtime), then invoked python on it (updates pyc to that mtime), then that python invocation updates the same script a second time and those changes were weirdly lost and confused me for a while. after having to delete the pycache a few times, i looked into it and ended up at (seemingly) this issue.

also (not relevant), having not looked at the code, what happens in 2038 when 64-bit file timestamps go over the 32-bit value? will the pyc just wrap around or is there some other planned handling?

maybe this format should be updated to 64-bit milliseconds (this seems.. difficult, maybe a new magic?) (and technically milliseconds can also race, though python is usually not fast enough to hit that), or perhaps the pycache should be invalidated when the mtime matches (but this would loop a bunch of times when the script is ran multiple times on the same second until current time is 1 second later than the filetime)

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs
  • gh-121502

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Führe die Shell-Reproduktion unter CPython 3.12 aus und untersuche die mit dem Import-Cache verbundene .pyc-Invalidierung und Zeitstempelverarbeitung. Vergleiche das Verhalten mit dem verlinkten PR gh-121502; als abgeschlossen sollte gelten, dass wiederholte Bearbeitungen innerhalb derselben Sekunde nicht mehr veralteten Quelltext ausführen und die relevanten Randfälle abgedeckt sind.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
tooling
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.