sparklemotion / sparklemotion/sqlite3-ruby
Insert thousands or millions of rows in bulk
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 845
- Forks
- 220
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 3
Description
What is the fastest way to batch insert a large number of rows in SQLite using this gem?
Currently I use this:
db.execute('BEGIN TRANSACTION')
sql = 'INSERT INTO items (id, name) VALUES (?, ?)'
items.each_with_index { |name, id| db.execute(sql, [id, name]) }
db.execute('COMMIT')
The items come from a large files and need to be inserted in the SQLite database as fast as possible.
- Is there any better / optimized solution?
- For example I was considering to keep all the inserts in Ruby in an array and then use
execute_batch, however I can't find a way to sanitize / interpolate the statements in this case. Is there a method likedb.sanitize(sql, [id, name])?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue provides Ruby examples using db.execute, transactions, and execute_batch; start by reviewing those APIs and their parameter-binding behavior in the gem. The issue names no file or test, so first identify the relevant implementation and test entry points, then define the fastest supported bulk-insert approach and how completion should be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby, sqlite
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100