msgspec / msgspec/msgspec

CodSpeed: test_pickle_load[arm] is bimodal and randomly flags unrelated PRs

Open
#1,110 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.1k
Forks
193
Avg merge
3d 20h
Merged PRs (30d)
15

Description

test_pickle_load[arm] (added in #1092) is bimodal: it flips between ~1.6 ms and ~1.8 ms across commits that don't touch the pickle path at all, so with the 2% threshold it randomly paints unrelated PRs red.

Evidence from this week:

  • #978 (decimal_format encoder option, head 045a25c vs main b36a4a1): flagged -10.05% (1.6 -> 1.8 ms), check red. The diff cannot affect struct pickling.
  • #1105 (to_builtins frozendict fix, head 5a54d59 vs main 8f7ac06): same benchmark reported +10.28% (1.8 -> 1.6 ms), check green (improvements don't gate).
  • The baseline itself flipped between those two comparisons: main measured 1.6 ms at b36a4a1 (06-27) and 1.8 ms at 8f7ac06 (06-30), with exactly one commit in between (#1052, frozendict) that doesn't touch the pickle path either.

Since this is simulation mode (instruction counting), it's not run-to-run measurement noise: the benchmark appears sensitive to code layout/alignment in _core.c, so any code addition can toggle it between the two states. Related to the general variance discussion in #1075 and CodSpeedHQ/codspeed#409.

Options (all live in the CodSpeed dashboard, admin only - there's no repo-side config for this):

  1. Set a per-benchmark regression threshold for test_pickle_load[arm] above the flip amplitude (say 15%). Keeps measuring, stops the false gating.
  2. Ignore the benchmark entirely in CodSpeed.
  3. The repo-wide "informational status check" toggle, which would also address the "CI looks perpetually broken" complaint from #1097.

Option 1 seems the most targeted. Worth checking whether the sibling test_pickle_* benchmarks from #1092 show the same behavior before picking.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the CodSpeed dashboard with test_pickle_load[arm] and compare the sibling test_pickle_* benchmarks added in #1092. Review the variance context in #1075 and CodSpeedHQ/codspeed#409, then confirm whether a per-benchmark threshold above the observed flip is appropriate. Done means the false regression gating is addressed through the selected dashboard setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.