examples/benchmarks_dynamic/workflow.py run_all has a AssertionError: NaN loss!

🐛 Bug Description

To Reproduce

Steps to reproduce the behavior:

1. download the repo with git clone https://github.com/microsoft/qlib.git
2. run python setup.py build_ext --inplace as suggested in Q&A section 3.
3. run python examples/benchmarks_dynamic/DDG-DA/workflow.py run_all on project path.

Screenshot

2022-11-16 14:34:58.594 | WARNING | qlib.tests.data:qlib_data:150 - Data already exists: ~/.qlib/qlib_data/cn_data, the data download will be skipped
If downloading is required: exists_skip=False or change target_dir
[131822:MainThread](2022-11-16 14:34:58,601) INFO - qlib.Initialization - [config.py:416] - default_conf: client.
[131822:MainThread](2022-11-16 14:34:58,633) INFO - qlib.workflow - [expm.py:31] - experiment manager uri is at file:/home/pengqlu/code/qlib/qlib/new_qlib/qlib/mlruns
[131822:MainThread](2022-11-16 14:34:58,634) INFO - qlib.Initialization - [init.py:74] - qlib successfully initialized based on client settings.
[131822:MainThread](2022-11-16 14:34:58,634) INFO - qlib.Initialization - [init.py:76] - data_path={'__DEFAULT_FREQ': PosixPath('/home/pengqlu/.qlib/qlib_data/cn_data')}
[131822:MainThread](2022-11-16 14:35:10,601) INFO - qlib.timer - [log.py:117] - Time cost: 11.901s | Loading data Done
[131822:MainThread](2022-11-16 14:35:11,567) INFO - qlib.timer - [log.py:117] - Time cost: 0.235s | DropnaLabel Done
/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/data/dataset/processor.py:316: FutureWarning: Not prepending group keys to the result index of transform-like apply. In the future, the group keys will be included in the index, regardless of whether the applied function returns a like-indexed object.
To preserve the previous behavior, use

    >>> .groupby(..., group_keys=False)

To adopt the future behavior and silence this warning, use

    >>> .groupby(..., group_keys=True)

df[cols] = df[cols].groupby("datetime").apply(self.zscore_func)
/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/data/dataset/processor.py:316: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df[cols] = df[cols].groupby("datetime").apply(self.zscore_func)
[131822:MainThread](2022-11-16 14:35:15,812) INFO - qlib.timer - [log.py:117] - Time cost: 4.245s | CSZScoreNorm Done
[131822:MainThread](2022-11-16 14:35:15,814) INFO - qlib.timer - [log.py:117] - Time cost: 5.213s | fit & process data Done
[131822:MainThread](2022-11-16 14:35:15,815) INFO - qlib.timer - [log.py:117] - Time cost: 17.115s | Init data Done
[131822:MainThread](2022-11-16 14:35:19,244) INFO - qlib.workflow - [expm.py:316] - <mlflow.tracking.client.MlflowClient object at 0x2b9826fb2f70>
[131822:MainThread](2022-11-16 14:35:19,251) WARNING - qlib.workflow - [expm.py:196] - No valid experiment found. Create a new experiment with name feature_importance.
[131822:MainThread](2022-11-16 14:35:19,277) INFO - qlib.workflow - [exp.py:260] - Experiment 454086792307167571 starts running ...
[131822:MainThread](2022-11-16 14:35:19,719) INFO - qlib.workflow - [recorder.py:339] - Recorder 9ec4e5d2442f4833a1219966d5f86d6d starts running under Experiment 454086792307167571 ...
ModuleNotFoundError. CatBoostModel are skipped. (optional: maybe installing CatBoostModel can fix it.)
ModuleNotFoundError. XGBModel is skipped(optional: maybe installing xgboost can fix it).
Training until validation scores don't improve for 50 rounds
[20] train's l2: 0.959367 valid's l2: 0.992761
[40] train's l2: 0.941031 valid's l2: 0.996238
[60] train's l2: 0.92202 valid's l2: 0.999542
Early stopping, best iteration is:
[12] train's l2: 0.96859 valid's l2: 0.992723
[131822:MainThread](2022-11-16 14:35:28,209) INFO - qlib.timer - [log.py:117] - Time cost: 1.484s | waiting async_log Done
[131822:MainThread](2022-11-16 14:35:43,440) INFO - qlib.timer - [log.py:117] - Time cost: 14.674s | Loading data Done
/data/pengqlu/miniconda3/envs/gnn/lib/python3.8/site-packages/numpy/lib/nanfunctions.py:1095: RuntimeWarning: All-NaN slice encountered
result = np.apply_along_axis(_nanmedian1d, axis, a, overwrite_input)
[131822:MainThread](2022-11-16 14:36:08,276) INFO - qlib.timer - [log.py:117] - Time cost: 24.161s | RobustZScoreNorm Done
[131822:MainThread](2022-11-16 14:36:09,065) INFO - qlib.timer - [log.py:117] - Time cost: 0.788s | Fillna Done
[131822:MainThread](2022-11-16 14:36:09,520) INFO - qlib.timer - [log.py:117] - Time cost: 0.237s | DropnaLabel Done
/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/data/dataset/processor.py:352: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df[cols] = t
[131822:MainThread](2022-11-16 14:36:09,930) INFO - qlib.timer - [log.py:117] - Time cost: 0.409s | CSRankNorm Done
[131822:MainThread](2022-11-16 14:36:09,932) INFO - qlib.timer - [log.py:117] - Time cost: 26.491s | fit & process data Done
[131822:MainThread](2022-11-16 14:36:09,932) INFO - qlib.timer - [log.py:117] - Time cost: 41.166s | Init data Done
examples/benchmarks_dynamic/DDG-DA/workflow.py:85: FutureWarning: Not prepending group keys to the result index of transform-like apply. In the future, the group keys will be included in the index, regardless of whether the applied function returns a like-indexed object.
To preserve the previous behavior, use

    >>> .groupby(..., group_keys=False)

To adopt the future behavior and silence this warning, use

    >>> .groupby(..., group_keys=True)

feature_selected = feature_selected.groupby("datetime").apply(lambda df: (df - df.mean()).div(df.std()))
[131822:MainThread](2022-11-16 14:36:19,938) INFO - qlib.timer - [log.py:117] - Time cost: 0.039s | Loading data Done
[131822:MainThread](2022-11-16 14:36:19,939) INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | fit & process data Done
[131822:MainThread](2022-11-16 14:36:19,939) INFO - qlib.timer - [log.py:117] - Time cost: 0.040s | Init data Done
[131822:MainThread](2022-11-16 14:36:20,828) WARNING - qlib.data - [data.py:662] - load calendar error: freq=day, future=True; return current calendar!
[131822:MainThread](2022-11-16 14:36:20,829) WARNING - qlib.data - [data.py:665] - You can get future calendar by referring to the following document: https://github.com/microsoft/qlib/blob/main/scripts/data_collector/contrib/README.md
[131822:MainThread](2022-11-16 14:36:20,864) WARNING - qlib.workflow - [expm.py:196] - No valid experiment found. Create a new experiment with name data_sim_s20.
train tasks: 0%| | 0/151 [00:00<?, ?it/s][131822:MainThread](2022-11-16 14:36:20,894) INFO - qlib.workflow - [expm.py:316] - <mlflow.tracking.client.MlflowClient object at 0x2b984458fbe0>
[131822:MainThread](2022-11-16 14:36:20,906) INFO - qlib.workflow - [exp.py:260] - Experiment 741558927160458500 starts running ...
[131822:MainThread](2022-11-16 14:36:21,024) INFO - qlib.workflow - [recorder.py:339] - Recorder 2694190356ac4aa1b10b7ae6b859b6e4 starts running under Experiment 741558927160458500 ...
[131822:MainThread](2022-11-16 14:36:26,936) INFO - qlib.workflow - [record_temp.py:194] - Signal record 'pred.pkl' has been saved as the artifact of the Experiment 741558927160458500
'The following are prediction results of the LinearModel model.'
score
datetime instrument
2008-01-02 SH600000 0.350154
SH600004 0.379545
SH600006 -0.069562
SH600007 -0.316895
SH600008 -0.075854
[131822:MainThread](2022-11-16 14:36:27,516) INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | waiting async_log Done
train tasks: 1%|▎ | 1/151 [00:06<16:34, 6.63s/it][131822:MainThread](2022-11-16 14:36:27,526) INFO - qlib.workflow - [expm.py:316] - <mlflow.tracking.client.MlflowClient object at 0x2b984458f250>
[131822:MainThread](2022-11-16 14:36:27,538) INFO - qlib.workflow - [exp.py:260] - Experiment 741558927160458500 starts running ...
[131822:MainThread](2022-11-16 14:36:27,642) INFO - qlib.workflow - [recorder.py:339] - Recorder be056bf7d1094a7b85dde96f032933f9 starts running under Experiment 741558927160458500 ...
[131822:MainThread](2022-11-16 14:36:28,386) INFO - qlib.workflow - [record_temp.py:194] - Signal record 'pred.pkl' has been saved as the artifact of the Experiment 741558927160458500
'The following are prediction results of the LinearModel model.'
score
datetime instrument
2008-01-02 SH600000 -1.029721
SH600004 -0.384231
SH600006 0.185993
SH600007 -0.410950
SH600008 -0.634558
...
...
...
score
datetime instrument
2008-01-02 SH600000 -0.600870
SH600004 0.898989
SH600006 -1.348339
SH600007 -0.543934
SH600008 0.136504
[131822:MainThread](2022-11-16 14:39:00,969) INFO - qlib.timer - [log.py:117] - Time cost: 0.101s | waiting async_log Done
train tasks: 100%|█████████████████████████████████████| 151/151 [02:40<00:00, 1.06s/it]
calc: 0%| | 0/151 [00:00<?, ?it/s]/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/workflow/recorder.py:425: FutureWarning: mlflow.tracking.client.MlflowClient.download_artifacts is deprecated since 2.0. This method will be removed in a future release. Use mlflow.artifacts.download_artifacts instead.
path = self.client.download_artifacts(self.id, name)
calc: 100%|████████████████████████████████████████████| 151/151 [00:05<00:00, 28.80it/s]
[131822:MainThread](2022-11-16 14:39:30,016) INFO - qlib.MetaDatasetDS - [dataset.py:278] - Example task for training meta model: {'dataset': {'class': 'qlib.data.dataset.DatasetH', 'kwargs': {'handler': 'file:///home/pengqlu/code/qlib/qlib/new_qlib/qlib/examples/benchmarks_dynamic/DDG-DA/handler_proxy.pkl', 'segments': {'train': (Timestamp('2008-01-02 00:00:00'), Timestamp('2010-12-03 00:00:00')), 'test': (Timestamp('2011-01-04 00:00:00'), Timestamp('2011-01-31 00:00:00'))}}}}
creating meta tasks: 100%|█████████████████████████████| 115/115 [01:24<00:00, 1.35it/s]
[131822:MainThread](2022-11-16 14:40:54,966) INFO - qlib.workflow - [expm.py:316] - <mlflow.tracking.client.MlflowClient object at 0x2b984402a160>
[131822:MainThread](2022-11-16 14:40:54,989) WARNING - qlib.workflow - [expm.py:196] - No valid experiment found. Create a new experiment with name DDG-DA.
[131822:MainThread](2022-11-16 14:40:55,017) INFO - qlib.workflow - [exp.py:260] - Experiment 353349595228998037 starts running ...
[131822:MainThread](2022-11-16 14:40:55,136) INFO - qlib.workflow - [recorder.py:339] - Recorder 42f13b02da134b839dc616e3560f8f01 starts running under Experiment 353349595228998037 ...
[131822:MainThread](2022-11-16 14:41:04,932) INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | waiting async_log Done
[131822:MainThread](2022-11-16 14:41:04,933) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[AssertionError: NaN loss!].
File "examples/benchmarks_dynamic/DDG-DA/workflow.py", line 262, in
fire.Fire(DDGDA)
File "/data/pengqlu/miniconda3/envs/gnn/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/data/pengqlu/miniconda3/envs/gnn/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/data/pengqlu/miniconda3/envs/gnn/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "examples/benchmarks_dynamic/DDG-DA/workflow.py", line 252, in run_all
self.train_meta_model()
File "examples/benchmarks_dynamic/DDG-DA/workflow.py", line 177, in train_meta_model
mm.fit(md)
File "/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/contrib/meta/data_selection/model.py", line 154, in fit
self.run_epoch(f"{phase}_noweight", task_list, 0, opt, {}, ignore_weight=True)
File "/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/contrib/meta/data_selection/model.py", line 96, in run_epoch
assert not np.isnan(loss.detach().item()), "NaN loss!"
AssertionError: NaN loss!

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

• Qlib version: I did not install Qlib at all.
• Python version: 3.8
• OS (Windows, Linux, MacOS): linux
• Commit number (optional, please provide it if you are using the dev version):
  commit b51e881be3075c72f6fce30643a39b368a76e368
  Author: Maxim Smolskiy mithridatus@mail.ru
  Date: Tue Nov 15 03:49:36 2022 +0300

Additional Notes

Start by running python examples/benchmarks_dynamic/DDG-DA/workflow.py run_all after the setup steps in the report. Read examples/benchmarks_dynamic/DDG-DA/workflow.py, especially the workflow section around line 262 and the DDG-DA training stages preceding the NaN loss! assertion. Done means the benchmark completes without the NaN-loss assertion.