python / python/cpython

SQLite3 multithreading cache inconsistency

Aperta
#118,172 10 commenti 9 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

extension-modules topic-sqlite3 type-bug
Lingua principale
Python
Stelle
77.2k
Fork
36k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

Bug report

Bug description:

When using SQLite3 in multi-threading application, the fetch reuslts are not consistent. After some testing, this seems to be caused by the caching.
The SQLite threading mode is serialized.
This issue only exists in python 3.12 and 3.13. No issue in 3.11.
Currently solution is to set cached_statements=0 when connect to the database.

A simple demo to reproduce the error:

import sqlite3
import threading

KB = sqlite3.connect(
    "example.db", check_same_thread=False
)

def execute_query():
    cursor = KB.cursor()
    cursor.execute("SELECT * FROM test_table")
    result = cursor.fetchall()
    assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
    return result

def run_threads():
    threads = []
    for i in range(50):
        thread = threading.Thread(target=execute_query)
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()

def test_multithreading():
    cursor = KB.cursor()

    cursor.execute(
        """CREATE TABLE IF NOT EXISTS test_table (id INTEGER PRIMARY KEY, value TEXT)"""
    )
    KB.commit()

    cursor.execute("""DELETE FROM test_table""")
    KB.commit()

    cursor.execute("""INSERT INTO test_table (value) VALUES ('test1')""")
    cursor.execute("""INSERT INTO test_table (value) VALUES ('test2')""")
    cursor.execute("""INSERT INTO test_table (value) VALUES ('test3')""")
    KB.commit()

    run_threads()

    KB.close()

if __name__ == "__main__":
    test_multithreading()

Test output: (Different in each run because of inconsistent fetch results)

Exception in thread Thread-3 (execute_query):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
Exception in thread Thread-7 (execute_query):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self._target(*self._args, **self._kwargs)
  File "/home/user/tmp/kb_test.py", line 13, in execute_query
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
Exception in thread Thread-12 (execute_query):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
    self._target(*self._args, **self._kwargs)
AssertionError: [(1, 'test2'), (3, 'test3')]
  File "/home/user/tmp/kb_test.py", line 13, in execute_query
Exception in thread Thread-1 (execute_query):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
    assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
    self.run()
AssertionError: [(3, 'test3')]
  File "/usr/lib/python3.12/threading.py", line 1010, in run
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/tmp/kb_test.py", line 13, in execute_query
    self._target(*self._args, **self._kwargs)
  File "/home/user/tmp/kb_test.py", line 13, in execute_query
    assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
    assert result == [(1, 'test1'), (2, 'test2'), (3, 'test3')], str(result)
AssertionError: [(None, None), (), (2, 'test2'), (), (), (2, 'test2'), (0, None), (1, 'test1'), (2, 'test2'), (3, 'test3')]
AssertionError: [(3, 'test3')]
CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia con il riproduttore Python fornito per sqlite3.connect(), confrontando Python 3.11 con 3.12 e 3.13 mentre esegui la query con 50 thread e vari cached_statements. Analizza il comportamento della cache delle istruzioni di sqlite3 nella modalità di threading serializzata di SQLite. Il lavoro è completato quando il recupero con i thread restituisce costantemente le tre righe previste senza richiedere cached_statements=0.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, sqlite
Ambito
databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.