Running mypy on script with pd.pivot_table gets exponentially slower with more columns
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20.6k
- Forks
- 3.3k
- PR merge metrics
- PR metrics pending
Description
Bug Report
When running mypy on a script containing pd.pivot_table, the amount of time for mypy to finish grows exponentially with the number of columns selected and aggregated in the table.
To Reproduce
from string import ascii_letters
import numpy as np
import pandas as pd
df = pd.DataFrame({letter: [1, 2, 3] for letter in ascii_letters[:14]})
df2 = pd.pivot_table(
df,
values=["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n"],
columns=["a"],
aggfunc={
"a": np.sum,
"b": np.sum,
"c": np.sum,
"d": np.sum,
"e": np.sum,
"f": np.sum,
"g": np.sum,
"h": np.sum,
"i": np.sum,
"j": np.sum,
"k": np.sum,
"l": np.sum,
"m": np.sum,
"n": np.sum,
},
)
Expected Behavior
It should finish in <1s, like it does if there are fewer columns
Actual Behavior
Takes 200 seconds to run mypy on this script
Your Environment
- Mypy version used: 1.7.0
- Mypy command-line flags: None
- Mypy configuration options from
mypy.ini(and other config files): None - Python version used: 3.8.15
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Python reproducer with mypy 1.7.0 and compare runtime as the selected columns increase. Profile the type-checking path to identify the source of the growth; done means the reproducer completes in under one second with the larger column set without changing its behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- devtools, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100