python / python/mypy

Sqlite writes limit parallel scaling

Open
#21,235 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

parallel checking performance
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

I measured time spent in sqlite operations when running the mypy_parallel benchmark. When using 8 workers, sqlite writes were over 10x slower compared to using just 1 worker, likely because only one worker can be writing at any time. This seems to be one of the main sources of limited parallel scaling on macOS at least.

Here are some ideas which might help:

  1. Instead of writing cache files in each worker, send them via IPC and have the main process write everything.
  2. Use a separate extra cache file per worker, and the main process will copy per-worker cached data to the main database. Every worker reads from the main database.
  3. Use a separate extra cache file per worker, and somehow allow each worker to read from the right database but always write to their own database.
  4. Shard the database -- have 1 (non-exclusive) database per worker, and use module name based hash to choose the target database for both reading and writing.

Next steps:

  • Measure on Linux as well
  • Prototype some of the ideas and measure impact (at least ones that are easy enough to implement)

cc @ilevkivskyi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the mypy_parallel benchmark and measuring SQLite write time with one and eight workers on macOS and Linux. Then review the four proposed cache-writing approaches and prototype the easiest candidates. Done means reporting comparable measurements and the impact on parallel scaling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.