yesodweb / yesodweb/persistent
putMany is very expensive
Nobody has claimed this yet.
- Dominant language
- Haskell
- Stars
- 486
- Forks
- 306
- PR merge metrics
- No merged PRs in 30d
Description
We have a Postgres db which is filled doing bulk inserts and reaches up to 30GB. We used putMany in chunks of 1000 rows for this. We noticed that this check https://github.com/yesodweb/persistent/blob/master/persistent/Database/Persist/Sql/Orphan/PersistUnique.hs#L70 adds a very heavy toll and removing it makes everything drastically faster
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read persistent/Database/Persist/Sql/Orphan/PersistUnique.hs at line 70 and trace how the check is reached by putMany. Reproduce the reported bulk-insert workload against PostgreSQL, compare performance with and without the check, and identify the behavior-preserving change needed to reduce its cost. Done means the relevant behavior remains covered and the bulk-insert overhead is demonstrably lower.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell, postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100