microsoft / microsoft/qlib

Anyone successfully run a US benchmark yet? There seem to be a bug with US version, instrument variable got evaluated to NAN.

Open
#1,428 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

🐛 Bug Description

To Reproduce

Steps to reproduce the behavior:

  1. take any yaml file under examples/benchmark. I'll take workflow_config_lightgbm_Alpha158.yaml for example.

  2. Change all fields from china version to US version:
    2.1 change (provider_uri: "/.qlib/qlib_data/cn_data") to (provider_uri: "/.qlib/qlib_data/us_data")
    2.2 change( region: cn) to (region: us)
    2.3 change (market: &market csi300) to (market: &market sp500)
    2.4 change (benchmark: &benchmark SH000300) to (benchmark: &benchmark ^GSPC)

  3. Run
    qrun workflow_config_lightgbm_Alpha158.yaml

Expected Behavior

AttributeError: 'float' object has no attribute 'lower'

Upon closer debugging, the instrument variable was evaluated to Nan.

Screenshot

image

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

Windows
AMD64
Windows-10-10.0.22621-SP0
10.0.22621

Python version: 3.8.15 (default, Nov 24 2022, 14:38:14) [MSC v.1916 64 bit (AMD64)]

Qlib version: 0.9.0.99
numpy==1.23.5
pandas==1.5.2
scipy==1.9.3
requests==2.28.1
sacred==0.8.2
python-socketio==5.7.2
redis==4.4.0
python-redis-lock==4.0.0
schedule==1.1.0
cvxpy==1.2.3
hyperopt==0.1.2
fire==0.5.0
statsmodels==0.13.5
xlrd==2.0.1
plotly==5.11.0
matplotlib==3.6.2
tables==3.8.0
pyyaml==6.0
mlflow==1.30.0
tqdm==4.64.1
loguru==0.6.0
lightgbm==3.3.3
tornado==6.2
joblib==1.2.0
fire==0.5.0
ruamel.yaml==0.17.21

Additional Notes

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/benchmark/workflow_config_lightgbm_Alpha158.yaml and reproduce the US substitutions using qrun. Trace how the US provider, region, market, and benchmark values are loaded until instrument becomes NaN and the lower() error occurs. Done means the US benchmark runs without the NaN instrument or AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, fintech-quant
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.