SQLite3 multithreading cache inconsistency
未关闭
还没有人认领这个 Issue。
extension-modules
topic-sqlite3
type-bug
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 36k
- PR 合并指标
- PR 指标待抓取
描述
Bug report
Bug description:
When using SQLite3 in multi-threading application, the fetch reuslts are not consistent. After some testing, this seems to be caused by the caching.
The SQLite threading mode is serialized.
This issue only exists in python 3.12 and 3.13. No issue in 3.11.
Currently solution is to set cached_statements=0 when connect to the database.
A simple demo to reproduce the error:
import sqlite3
import threading
KB = sqlite3.connect(
"example.db", check_same_thread=False
)
def execute_query():
cursor = KB.cursor()
cursor.execute("SELECT * FROM test_table")
result = cursor.fetchall()
assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
return result
def run_threads():
threads = []
for i in range(50):
thread = threading.Thread(target=execute_query)
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
def test_multithreading():
cursor = KB.cursor()
cursor.execute(
"""CREATE TABLE IF NOT EXISTS test_table (id INTEGER PRIMARY KEY, value TEXT)"""
)
KB.commit()
cursor.execute("""DELETE FROM test_table""")
KB.commit()
cursor.execute("""INSERT INTO test_table (value) VALUES ('test1')""")
cursor.execute("""INSERT INTO test_table (value) VALUES ('test2')""")
cursor.execute("""INSERT INTO test_table (value) VALUES ('test3')""")
KB.commit()
run_threads()
KB.close()
if __name__ == "__main__":
test_multithreading()
Test output: (Different in each run because of inconsistent fetch results)
Exception in thread Thread-3 (execute_query):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
Exception in thread Thread-7 (execute_query):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self._target(*self._args, **self._kwargs)
File "/home/user/tmp/kb_test.py", line 13, in execute_query
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
Exception in thread Thread-12 (execute_query):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
self._target(*self._args, **self._kwargs)
AssertionError: [(1, 'test2'), (3, 'test3')]
File "/home/user/tmp/kb_test.py", line 13, in execute_query
Exception in thread Thread-1 (execute_query):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
self.run()
AssertionError: [(3, 'test3')]
File "/usr/lib/python3.12/threading.py", line 1010, in run
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/home/user/tmp/kb_test.py", line 13, in execute_query
self._target(*self._args, **self._kwargs)
File "/home/user/tmp/kb_test.py", line 13, in execute_query
assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
AssertionError: [(None, None), (), (2, 'test2'), (), (), (2, 'test2'), (0, None), (1, 'test1'), (2, 'test2'), (3, 'test3')]
AssertionError: [(3, 'test3')]
CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Linux
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从提供的 Python sqlite3.connect() 复现程序开始,在运行 50 线程查询并改变 cached_statements 的同时,对比 Python 3.11、3.12 和 3.13。调查 serialized SQLite 线程模式下 sqlite3 语句缓存的行为。当多线程获取能够稳定返回预期的三行,且不需要 cached_statements=0 时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python, sqlite
- 领域
- databases
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 35/100