duckdb / duckdb/duckdb-java

FATAL Error: Failed to create checkpoint because of error...

Aperta
#206 18 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C++
Stelle
127
Fork
80
Merge medio
13h 49m
PR unite (30g)
48

Descrizione

JAVA 17, DuckDB 1.2 (duckdb_jdbc-1.2.1.jar)

I am trying to perform some queued INSERTs (specifically preparedStatement, batch inserts) using CompletableFuture (i.e. background executor, but same process). All these go to the same table.

The background method uses (partially) :
`try (var connection = duckDBConnection.duplicate();
var statement = connection.prepareStatement(INSERT_TEMPLATE))) {
for (...) {
statement.setString();
statement.set...
statement.addBatch();
...
}
statement.executeBatch();
} `

I'm using CompletableFuture.runAsync, so these go on the Java commonPool() executor, queued to run asap.

After a few of these complete, I'm getting the exception:
`java.sql.SQLException:` FATAL Error: Failed to create checkpoint because of error: Failed to delete file "e:\vectorindex\efr.duckdb.wal": The process cannot access the file because it is being used by another process.

at org.duckdb.DuckDBNative.duckdb_jdbc_execute(Native Method)
at org.duckdb.DuckDBPreparedStatement.execute(DuckDBPreparedStatement.java:148)
at org.duckdb.DuckDBPreparedStatement.executeBatchedPreparedStatement(DuckDBPreparedStatement.java:489)
at org.duckdb.DuckDBPreparedStatement.executeBatch(DuckDBPreparedStatement.java:474)
...`

This is entirely unexpected since all of the code is running in the same process, and according to the docs, this should be fine. And they are all unique INSERTs (there isn't even a PK on the table), so there should be no row-based table contention.

It appears that DuckDB is performing checkpoints on its own...maybe when the duplicate session is closing in the method? And whether it is performing checkpoints or not, this appears to be a bug, because it's documented that multiple threads within the same process should be able to r/w the same database.

Does the WAL strategy need to be adjusted (internally)?

Are there any workarounds for now?

Thanks!

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Riproduci il carico di lavoro segnalato con Java 17, DuckDB 1.2.1, connessioni duplicate, CompletableFuture.runAsync e INSERT preparati eseguiti in batch. Inizia da DuckDBPreparedStatement.executeBatch e dall’entry nativa duckdb_jdbc_execute nello stack trace, quindi traccia la chiusura delle connessioni e il comportamento del checkpoint. Il lavoro è completato quando è stata identificata la causa del fallimento dell’eliminazione del WAL e sono stati documentati un fix o un workaround verificati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java, sql
Ambito
databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.