microsoft / microsoft/qlib

Potential performance issue: concat slow in pandas below 2.1 version

Open
#1,753 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

Issue Description:

Hello.
I have discovered a performance degradation in the .concat function of pandas version below 2.1. And I notice some parts of the repository depend on pandas below 2.1 such as examples/benchmarks/TRA/requirements.txt, examples/benchmarks/Sandwich/requirements.txt. I am not sure whether this performance problem in pandas will affect this repository. I found some discussions on pandas GitHub related to this issue, including #50652 and #52685.
I also found that examples/benchmarks/TRA/src/model.py, examples/workflow_by_code.ipynb, and examples/benchmarks/TRA/Reports.ipynb used the influenced api. There may be more files using the influenced api.

Suggestion

I would recommend considering an upgrade to a different version of pandas >= 2.1 or exploring other solutions to optimize the performance of .concat.
Any other workarounds or solutions would be greatly appreciated.
Thank you!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the pandas pins in examples/benchmarks/TRA/requirements.txt and examples/benchmarks/Sandwich/requirements.txt, then inspect concat usage in examples/benchmarks/TRA/src/model.py, examples/workflow_by_code.ipynb, and examples/benchmarks/TRA/Reports.ipynb. Compare the affected workflows with pandas versions below and at 2.1. Done requires identifying the affected usage and documenting or implementing a confirmed upgrade or optimization path.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.