sqlalchemy / sqlalchemy/alembic

add this recipe for large bulk_inserts to the bulk_insert docs

Open
#144 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Python
Stars
4.4k
Forks
375
PR merge metrics
No merged PRs in 30d

Description

Migrated issue, originally created by Florian Rüchel (@javex)

I recently had a problem where MySQL would not like to bulk_insert 80k rows at once. Instead I had to feed it 10k chunks. So I split the list into 10k chunks and gave each to alembic.

I don't know if it would make any sense, but I'd suggest adding a parameter chunk_size to bulk_insert. For reference, I did it like this:

    for index, item in enumerate(data):
        if index % 10000 == 0:
            if index != 0:
                op.bulk_insert(table, tmpdata)
            tmpdata = []
        tmpdata.append(item)
if tmpdata:
    op.bulk_insert(GeoIP.__table__, tmpdata)

Let me know what you think of it. I'd be happy to create a patch for it, if you like the idea.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the bulk_insert documentation and the op.bulk_insert example in the issue, then review how the existing Alembic docs organize recipes. Done means documenting the 10,000-row chunking workaround clearly, including the final partial chunk and the MySQL motivation.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.