microsoft / microsoft/qlib

My question about ALSTM reproduction

Open
#1,765 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
48.7k
Forks
7.7k
No merged PRs in 30d

Description

❓ Questions and Help

(ALSTM-1) [steve@gpu01 ALSTM]$ qrun workflow_config_alstm_Alpha158.yaml 
[36025:MainThread](2024-03-19 21:56:20,365) INFO - qlib.Initialization - [config.py:416] - default_conf: client.
[36025:MainThread](2024-03-19 21:56:20,368) INFO - qlib.Initialization - [__init__.py:74] - qlib successfully initialized based on client settings.
[36025:MainThread](2024-03-19 21:56:20,368) INFO - qlib.Initialization - [__init__.py:76] - data_path={'__DEFAULT_FREQ': PosixPath('/home/steve/.qlib/qlib_data/cn_data')}
[36025:MainThread](2024-03-19 21:56:20,370) INFO - qlib.workflow - [exp.py:258] - Experiment 1 starts running ...
[36025:MainThread](2024-03-19 21:56:20,457) INFO - qlib.workflow - [recorder.py:341] - Recorder bae742011ad2426f9f0684c0cbeaba50 starts running under Experiment 1 ...
ModuleNotFoundError. CatBoostModel are skipped. (optional: maybe installing CatBoostModel can fix it.)
ModuleNotFoundError. XGBModel is skipped(optional: maybe installing xgboost can fix it).
[36025:MainThread](2024-03-19 21:56:22,388) INFO - qlib.ALSTM - [pytorch_alstm_ts.py:63] - ALSTM pytorch version...
[36025:MainThread](2024-03-19 21:56:22,579) INFO - qlib.ALSTM - [pytorch_alstm_ts.py:81] - ALSTM parameters setting:
d_feat : 20
hidden_size : 64
num_layers : 2
dropout : 0.0
n_epochs : 200
lr : 0.001
metric : loss
batch_size : 800
early_stop : 10
optimizer : adam
loss_type : mse
device : cuda:0
n_jobs : 20
use_GPU : True
seed : None
[36025:MainThread](2024-03-19 21:56:22,586) INFO - qlib.ALSTM - [pytorch_alstm_ts.py:126] - model:
ALSTMModel(
  (net): Sequential(
    (fc_in): Linear(in_features=20, out_features=64, bias=True)
    (act): Tanh()
  )
  (rnn): GRU(64, 64, num_layers=2, batch_first=True)
  (fc_out): Linear(in_features=128, out_features=1, bias=True)
  (att_net): Sequential(
    (att_fc_in): Linear(in_features=64, out_features=32, bias=True)
    (att_dropout): Dropout(p=0.0, inplace=False)
    (att_act): Tanh()
    (att_fc_out): Linear(in_features=32, out_features=1, bias=False)
    (att_softmax): Softmax(dim=1)
  )
)
[36025:MainThread](2024-03-19 21:56:22,587) INFO - qlib.ALSTM - [pytorch_alstm_ts.py:127] - model size: 0.0510 MB
[36025:MainThread](2024-03-19 21:56:25,678) INFO - qlib.timer - [log.py:127] - Time cost: 0.000s | waiting `async_log` Done
[36025:MainThread](2024-03-19 21:56:25,681) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[TypeError: cannot do slice indexing on Index with these indexers [1892] of type int].
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/bin/qrun", line 8, in <module>
    sys.exit(run())
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/workflow/cli.py", line 117, in run
    fire.Fire(workflow)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/workflow/cli.py", line 111, in workflow
    recorder = task_train(config.get("task"), experiment_name=experiment_name)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/model/trainer.py", line 127, in task_train
    _exe_task(task_config)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/model/trainer.py", line 46, in _exe_task
    dataset: Dataset = init_instance_by_config(task_config["dataset"], accept_types=Dataset)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/utils/mod.py", line 179, in init_instance_by_config
    return klass(**cls_kwargs, **kwargs)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/__init__.py", line 666, in __init__
    super().__init__(**kwargs)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/__init__.py", line 119, in __init__
    self.handler: DataHandler = init_instance_by_config(handler, accept_types=DataHandler)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/utils/mod.py", line 179, in init_instance_by_config
    return klass(**cls_kwargs, **kwargs)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/contrib/data/handler.py", line 173, in __init__
    super().__init__(
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 468, in __init__
    super().__init__(instruments, start_time, end_time, data_loader, **kwargs)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 100, in __init__
    self.setup_data()
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 610, in setup_data
    super().setup_data(**kwargs)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 144, in setup_data
    self._data = lazy_sort_index(self.data_loader.load(self.instruments, self.start_time, self.end_time))
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 135, in load
    {
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 136, in <dictcomp>
    grp: self.load_group_df(instruments, exprs, names, start_time, end_time, grp)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 217, in load_group_df
    df = D.features(instruments, exprs, start_time, end_time, freq=freq, inst_processors=inst_processors)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/data.py", line 1191, in features
    return DatasetD.dataset(instruments, fields, start_time, end_time, freq, inst_processors=inst_processors)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/data.py", line 924, in dataset
    data = self.dataset_processor(
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/qlib/data/data.py", line 578, in dataset_processor
    ParallelExt(n_jobs=workers, backend=C.joblib_backend, maxtasksperchild=C.maxtasksperchild)(task_l),
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/joblib/parallel.py", line 1952, in __call__
    return output if self.return_generator else list(output)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/joblib/parallel.py", line 1595, in _get_outputs
    yield from self._retrieve()
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/joblib/parallel.py", line 1699, in _retrieve
    self._raise_error_fast()
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/joblib/parallel.py", line 1734, in _raise_error_fast
    error_job.get_result(self.timeout)
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/joblib/parallel.py", line 736, in get_result
    return self._return_or_raise()
  File "/usr/local/Public_Anaconda3/anaconda3/envs/ALSTM-1/lib/python3.8/site-packages/joblib/parallel.py", line 754, in _return_or_raise
    raise self._result
TypeError: cannot do slice indexing on Index with these indexers [1892] of type int

Is it a problem about Dataframe?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the qrun workflow_config_alstm_Alpha158.yaml command and inspect the traceback path through qlib/data/dataset/loader.py and qlib/data/data.py. Trace the slice indexing failure around load_group_df and DatasetD.dataset, then document the data or indexing condition that causes it and verify the workflow completes or fails with a clearer explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.