duckdb / duckdb/duckdb-sqlite

Poor performance for simple operation on large-ish database.

Open
#129 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
291
Forks
45
Avg merge
10h 13m
Merged PRs (30d)
18

Description

### What happens?

I have a large-ish database. I am using libsql and not sqlite proper. It has two tables, one with 25M rows and one with 10M. The db file is ~50GB.

Then
```
time duckdb data.sqld/dbs/default/data "select * from message_queue limit 10"
real 0m46.187s
user 0m5.659s
sys 0m10.910s
```
The same query takes 60ms from sqlite.

Even just `show tables`
```
time duckdb data.sqld/dbs/default/data "show tables;"
┌─────────────────┐
│ name │
│ varchar │
├─────────────────┤
│ dicom │
│ message_queue │
│ sqlite_sequence │
└─────────────────┘
real 3m21.493s
user 3m29.433s
sys 0m10.378s
```
Using `SELECT name FROM sqlite_master WHERE type='table';` from a normal client 55ms.

This performance is very surprising. What is going on?

### To Reproduce

Can't really share much more.

### OS:

linux

### SQLite Version:

libsql 0.24.31

### DuckDB Version:

v1.1.3 19864453f7

### DuckDB Client:

cli

### Full Name:

JF Pambrun

### Affiliation:

See linkedin..

### Have you tried this on the latest `main` branch?

- [x] I agree

### Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

- [x] I agree

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.