pandas v0.24 performance issues
@WeiqiNs is already working on this.
Since Jun 5, 2019.
- Dominant language
- Python
- Stars
- 123
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
We noticed a serious time difference in our stats routines between the server and our local copies--as it turns out, this is a result of a pandas version difference.
Lines 197 and 203 of stats_model.py run as much as 10x slower on pandas v0.24.2 (which is on our dev boxes) compared to v0.23.4 (which is on our servers). Compare to line 200, which runs relatively fast on both versions--the addition of the eq() or ne() is the key difference.
here's some timing data (in seconds) that I grabbed on both versions:
0.24.2
getting labels: 2.9769787788391113
setting up data frame: 0.010141849517822266
tokens that appear once: 24.94732117652893
total # tokens: 1.0076451301574707
distinct # tokens: 21.80290699005127
average: 0.0008835792541503906
0.23.4
getting labels: 2.9242076873779297
setting up data frame: 0.005937099456787109
tokens that appear once: 2.862314224243164
total # tokens: 1.5265724658966064
distinct # tokens: 1.0599994659423828
average: 0.01018071174621582
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.