microsoft / microsoft/qlib

Cannot Run qrun with lightgbm workflow config

Open
#798 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

🐛 Bug Description

(qlib) 👍  examples   main  qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
[33800:MainThread](2022-01-03 15:22:20,670) INFO - qlib.Initialization - [config.py:391] - default_conf: client.
[33800:MainThread](2022-01-03 15:22:20,671) WARNING - qlib.Initialization - [config.py:416] - redis connection failed(host=127.0.0.1 port=6379), DiskExpressionCache and DiskDatasetCache will not be used!
[33800:MainThread](2022-01-03 15:22:20,672) INFO - qlib.Initialization - [init.py:68] - qlib successfully initialized based on client settings.
[33800:MainThread](2022-01-03 15:22:20,672) INFO - qlib.Initialization - [init.py:70] - data_path={'__DEFAULT_FREQ': PosixPath('/Users/waterking/.qlib/qlib_data/cn_data')}
[33800:MainThread](2022-01-03 15:22:20,673) INFO - qlib.workflow - [expm.py:282] - No tracking URI is provided. Use the default tracking URI.
[33800:MainThread](2022-01-03 15:22:20,673) INFO - qlib.workflow - [expm.py:318] - <mlflow.tracking.client.MlflowClient object at 0x7fa97d0b8ac0>
[33800:MainThread](2022-01-03 15:22:20,693) INFO - qlib.workflow - [exp.py:249] - Experiment 1 starts running ...
[33800:MainThread](2022-01-03 15:22:20,796) INFO - qlib.workflow - [recorder.py:290] - Recorder a3ea360a53a84f0dbb7dae7b7e683dcc starts running under Experiment 1 ...
/Users/waterking/opt/anaconda3/envs/qlib/lib/python3.8/site-packages/pyqlib-0.8.0.99-py3.8-macosx-10.9-x86_64.egg/qlib/utils/init.py:808: FutureWarning: MultiIndex.is_lexsorted is deprecated as a public function, users should use MultiIndex.is_monotonic_increasing instead.
if idx.is_monotonic_increasing and not (isinstance(idx, pd.MultiIndex) and not idx.is_lexsorted()):
[33800:MainThread](2022-01-03 15:23:21,552) INFO - qlib.timer - [log.py:113] - Time cost: 60.085s | Loading data Done
[33800:MainThread](2022-01-03 15:23:22,967) INFO - qlib.timer - [log.py:113] - Time cost: 0.411s | DropnaLabel Done
/Users/waterking/opt/anaconda3/envs/qlib/lib/python3.8/site-packages/pandas-1.3.5-py3.8-macosx-10.9-x86_64.egg/pandas/core/frame.py:3641: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self[k1] = value[k2]
[33800:MainThread](2022-01-03 15:23:27,797) INFO - qlib.timer - [log.py:113] - Time cost: 4.829s | CSZScoreNorm Done
[33800:MainThread](2022-01-03 15:23:27,835) INFO - qlib.timer - [log.py:113] - Time cost: 6.282s | fit & process data Done
[33800:MainThread](2022-01-03 15:23:27,836) INFO - qlib.timer - [log.py:113] - Time cost: 66.369s | Init data Done
[1] 33800 segmentation fault qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
/Users/waterking/opt/anaconda3/envs/qlib/lib/python3.8/site-packages/joblib-1.1.0-py3.8.egg/joblib/externals/loky/backend/resource_tracker.py:318: UserWarning: resource_tracker: There appear to be 2 leaked folder objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/Users/waterking/opt/anaconda3/envs/qlib/lib/python3.8/site-packages/joblib-1.1.0-py3.8.egg/joblib/externals/loky/backend/resource_tracker.py:333: UserWarning: resource_tracker: /var/folders/gr/17hy_zfj49l4p998g6xq2d340000gn/T/joblib_memmapping_folder_33800_95596af9eff64ff4b7116daedbbcbc53_a5376a7190bc40818f999d3d14e3d716: FileNotFoundError(2, 'No such file or directory')
warnings.warn('resource_tracker: %s: %r' % (name, e))
/Users/waterking/opt/anaconda3/envs/qlib/lib/python3.8/site-packages/joblib-1.1.0-py3.8.egg/joblib/externals/loky/backend/resource_tracker.py:333: UserWarning: resource_tracker: /var/folders/gr/17hy_zfj49l4p998g6xq2d340000gn/T/joblib_memmapping_folder_33800_ca9c83dbe40b49aeacc099c618d44dfa_9a4c5e1eb21743a884e50325b7cd17fa: FileNotFoundError(2, 'No such file or directory')
warnings.warn('resource_tracker: %s: %r' % (name, e))

To Reproduce

Steps to reproduce the behavior:

1.cd examples # Avoid running program under the directory contains qlib
2.qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

  • Qlib version: Newest
  • Python version: 3.8.5
  • OS (MacOS):
  • Commit number (optional, please provide it if you are using the dev version):

Additional Notes

Please help, thanks a lot!~

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure from the examples directory with qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml. Review that workflow configuration and run scripts/collect_info.py all to capture the missing environment details, then narrow down the stage causing the segmentation fault. Done means the workflow completes without the crash on the reported macOS setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.