mattn / mattn/go-sqlite3

Next and Scan appear to be relatively slow?

Open
#379 10 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted research
Dominant language
C
Stars
9.2k
Forks
1.2k
Avg merge
19m
Merged PRs (30d)
4

Description

Howdy. I'm not sure this really merits a Issues or if it would be better as a conversation some place. In using sqlite3 for a project and I noticed that it's quite a bit slower to select a large number of rows as compared to doing the same via their CLI client. ~4-5x slower. A little profiling shows that the time is mostly spent in Next and Scan. For small row sets it's not a big deal but I'm testing up to ~500K rows.

I suppose my starting question is how much overhead is expected going back and forth between Go and the C API? If it's expected to be fairly fast/low-overhead, any suspicions on where I might start looking?

Here's a sketch of approximately what the code is doing:

https://gist.github.com/mhat/ded97fe58922b4af9d03f375b465c339

There are two variations, one uses the new ScanTypes while the other is cheating with a list of string pointers. Neither appreciably moved the needly forScan.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked gist and profile the Next and Scan paths for large result sets, comparing the ScanTypes and string-pointer variations described in the issue. Investigate the Go-to-C API overhead and establish a reproducible comparison with the SQLite CLI; the issue does not define a specific fix or completion test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.