python / python/cpython

Uses a higher granularity than seconds for `.pyc` file invalidation

未關閉
#121,376 19 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

type-feature
主要語言
Python
星號
77.2k
分支
36k
PR 合併指標
PR 指標待擷取

描述

Bug report

Bug description:

reproduction:

#!/bin/sh 
 
mkdir -p src 
rm -fr src/__pycache__ 
rm -fr __pycache__ 
 
echo 'from src.test import f' > main.py 
echo 'print(f())' >> main.py 

echo 'def f():' > src/test.py 
echo '    return "Hello!"' >> src/test.py 
# prints "Hello!"
python3 main.py 

echo 'def f():' > src/test.py 
echo '    return "Goodbye!"' >> src/test.py 
# also prints "Hello!", but note the file does not say that anymore
python3 main.py

(this is probably sensitive to how fast your computer is)

this seems to stem from (i guess) that the timestamps are 32-bit, and write the current second (unix time, as 32-bit). and then, when the file is edited and the python script runs again, it gets a cache hit (same timestamp as file 32-bit mtime) and yields the old content.

is this known/intended? is there a mitigation that isn't just "delete the pycache"? i tried looking if anyone else ran into this and could not find anything.

i ran into this as part of a workflow that first edited a python script via shell (edits mtime), then invoked python on it (updates pyc to that mtime), then that python invocation updates the same script a second time and those changes were weirdly lost and confused me for a while. after having to delete the pycache a few times, i looked into it and ended up at (seemingly) this issue.

also (not relevant), having not looked at the code, what happens in 2038 when 64-bit file timestamps go over the 32-bit value? will the pyc just wrap around or is there some other planned handling?

maybe this format should be updated to 64-bit milliseconds (this seems.. difficult, maybe a new magic?) (and technically milliseconds can also race, though python is usually not fast enough to hit that), or perhaps the pycache should be invalidated when the mtime matches (but this would loop a bunch of times when the script is ran multiple times on the same second until current time is 1 second later than the filetime)

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs
  • gh-121502

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

在 CPython 3.12 上執行 shell 重現,並檢查 import cache 涉及的 .pyc 失效處理與時間戳處理。將行為與連結的 PR gh-121502 進行比較;應以重複的同秒編輯不再執行過時原始碼,且相關邊界情況已涵蓋為完成標準。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
tooling
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
需要釐清
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。