sparklemotion / sparklemotion/sqlite3-ruby

Insert thousands or millions of rows in bulk

Open
#343 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
845
Forks
220
Avg merge
2d 17h
Merged PRs (30d)
3

Description

What is the fastest way to batch insert a large number of rows in SQLite using this gem?

Currently I use this:

db.execute('BEGIN TRANSACTION')
sql = 'INSERT INTO items (id, name) VALUES (?, ?)'
items.each_with_index { |name, id| db.execute(sql, [id, name]) }
db.execute('COMMIT')

The items come from a large files and need to be inserted in the SQLite database as fast as possible.

  1. Is there any better / optimized solution?
  2. For example I was considering to keep all the inserts in Ruby in an array and then use execute_batch, however I can't find a way to sanitize / interpolate the statements in this case. Is there a method like db.sanitize(sql, [id, name])?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue provides Ruby examples using db.execute, transactions, and execute_batch; start by reviewing those APIs and their parameter-binding behavior in the gem. The issue names no file or test, so first identify the relevant implementation and test entry points, then define the fastest supported bulk-insert approach and how completion should be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby, sqlite
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.