python-poetry / python-poetry/tomlkit

key insertion is linear in size of document

Open
#540 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
850
Forks
162
Avg merge
13m
Merged PRs (30d)
2

Description

So inserting lots of keys is quadratic in the number of keys

#!/usr/bin/env python3

import gc
import time

import tomlkit


def insert_keys(n: int) -> float:
    doc = tomlkit.parse("[t]\n")
    table = doc["t"]

    gc.collect()
    gc.disable()
    try:
        start = time.perf_counter()
        for i in range(n):
            table[f"k{i}"] = i
        elapsed = time.perf_counter() - start
    finally:
        gc.enable()

    return elapsed


def main() -> None:
    print()
    print(f"{'N':>7} {'total (ms)':>12} {'us/key':>10} {'ratio':>7}")

    prev = None
    for n in (500, 1000, 2000, 4000, 8000):
        elapsed = insert_keys(n)
        total_ms = elapsed * 1e3
        us_per_key = elapsed / n * 1e6
        ratio = "" if prev is None else f"{elapsed / prev:6.2f}x"
        print(f"{n:>7} {total_ms:12.2f} {us_per_key:10.2f} {ratio:>7}")
        prev = elapsed


if __name__ == "__main__":
    main()

results

      N   total (ms)     us/key   ratio
    500        59.58     119.16
   1000       225.36     225.36   3.78x
   2000       983.92     491.96   4.37x
   4000      3569.23     892.31   3.63x
   8000     16247.27    2030.91   4.55x

in which each doubling of the number of keys causes the benchmark to take four times as long.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the inline Python benchmark against tomlkit's table key insertion path and inspect how each assignment updates the document. Trace the work that grows with the document size, then verify the fix by rerunning the benchmark and confirming that doubling the number of keys no longer makes total time roughly quadruple.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.