Running `run_all_models,py` cannot reproduce the benchmark results.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 48.7k
- Forks
- 7.7k
- PR merge metrics
- No merged PRs in 30d
Description
🐛 Bug Description
I ran python run_all_model.py run --models "lightgbm,mlp,gru,catboost,lstm" --dataset Alpha158
The results for 1 iteration. I know I only ran them for 1 iteration, but the results should be close if the code is right.
The result is quite different from benchmark reports
| Model Name | Dataset | IC | ICIR | Rank IC | Rank ICIR | Annualized Return | Information Ratio | Max Drawdown |
|---|---|---|---|---|---|---|---|---|
| CatBoost | Alpha158 | 0.0471±0.00 | 0.3352±0.00 | 0.0447±0.00 | 0.3328±0.00 | 0.0726±0.00 | 0.7846±0.00 | -0.1107±0.00 |
| MLP | Alpha158 | 0.0319±0.00 | 0.2459±0.00 | 0.0374±0.00 | 0.2825±0.00 | 0.0699±0.00 | 0.8855±0.00 | -0.1460±0.00 |
| GRU | Alpha158 | 0.0332±0.00 | 0.2580±0.00 | 0.0442±0.00 | 0.3596±0.00 | 0.0150±0.00 | 0.2140±0.00 | -0.1093±0.00 |
| LightGBM | Alpha158 | 0.0464±0.00 | 0.3779±0.00 | 0.0486±0.00 | 0.4038±0.00 | 0.1062±0.00 | 1.1867±0.00 | -0.1001±0.00 |
| LSTM | Alpha158 | 0.0282±0.00 | 0.2077±0.00 | 0.0438±0.00 | 0.3281±0.00 | 0.0268±0.00 | 0.3973±0.00 | -0.1197±0.00 |
Expected Behavior
Reproduce the benchmarks.
Environment
Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.
- Qlib version: Latest.
- Python version: 3.7.16
- OS: Linux
Additional Notes
None.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the run_all_model.py command and compare its one-iteration Alpha158 results for LightGBM, MLP, GRU, CatBoost, and LSTM with the benchmark reports linked in the issue. Run scripts/collect_info.py all if environment details are needed. Done means identifying and correcting the cause of the discrepancy so the command produces results close to the published benchmarks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100