duckdb / duckdb/duckdb-node

Connections don't run queries concurrently

Open
#16 5 comments 3 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
91
Forks
36
PR merge metrics
No merged PRs in 30d

Description

Not sure if this is WAD, a bug, or there is some other workaround here for doing connection pooling with DuckDB in nodejs...

Using an open database, I want to run some queries concurrently across different connections. Is there a way to accomplish this?

This test script shows that queries will not run concurrently, even if run on different connections:
```javascript
var duckdb = require('duckdb');

var db = new duckdb.Database(':memory:');

const connA = db.connect();
const connB = db.connect();

async function fastQuery() {
return new Promise((resolve, reject) => {
const t = Date.now();
connA.all(`select 1 from range(1,10)`, (err, res) => {
if(err) reject(err);
else {
resolve(Date.now() - t)
}
})
})
}

async function slowQuery() {
return new Promise((resolve, reject) => {
const t = Date.now();
connB.all(`select max(i) from (select 1 as i from range(1,10000000000));`, (err, res) => {
if(err) reject(err);
else {
resolve(Date.now() - t)
}
})
})
}

async function test() {
console.log("Run fast query");
console.log("Fast query time (ms): ", await fastQuery());

console.log("Run slow query");
console.log("Slow query time (ms): ", await slowQuery());

console.log("Run slow and fast query concurrently");
slowQuery().then(s => console.log("Slow query time (ms): ", s));
fastQuery().then(f => console.log("Fast query time (ms): ", f));
}

test();
```
The fast query should take a few milliseconds, while the slow query should take a few seconds. If the slow and fast queries are kicked off at the same time, even on different connections, the slow query blocks the fast query from executing. This is what I get when running the script above on an M2 MBP:
```
Run fast query
Fast query time (ms): 2
Run slow query
Slow query time (ms): 4386
Run slow and fast query concurrently
Slow query time (ms): 4701
Fast query time (ms): 4701
```

If I run the fast query on a completely different Database handle, then there is no problem running the fast query concurrently with the slow query:
```
Run fast query
Fast query time (ms): 3
Run slow query
Slow query time (ms): 4381
Run slow and fast query concurrently
Fast query on different DB handle (ms): 1
Slow query time (ms): 4700
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.