Request for data update #BUG ALSO

🌟 Request for data update

Convenient data retrieve method (like python -m qlib.run.get_data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn) can only get data to 2020-9-24. However, if we get data from yahoo manually, It costs lots of time(more than 8 hours here only for DOWNLOAD) and raise bugs.

And when it begins to normalize, it happens [58410:MainThread](2023-09-25 06:27:36,588) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[TypeError: can't compare offset-naive and offset-aware datetimes]. File "scripts/data_collector/yahoo/collector.py", line 1207, in <module> fire.Fire(Run) File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "scripts/data_collector/yahoo/collector.py", line 1182, in update_data_to_bin self.normalize_data_1d_extend(qlib_data_1d_dir) File "scripts/data_collector/yahoo/collector.py", line 1072, in normalize_data_1d_extend yc.normalize() File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/site-packages/qlib/scripts/data_collector/base.py", line 319, in normalize for _ in worker.map(self._executor, file_list): File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists for element in iterable: File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator yield fs.pop().result() File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/concurrent/futures/_base.py", line 444, in result return self.__get_result() File "/Users/bernoulli_hermes/opt/anaconda3/envs/qlibenv/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception TypeError: can't compare offset-naive and offset-aware datetimes.

Start with scripts/data_collector/yahoo/collector.py, especially update_data_to_bin and normalize_data_1d_extend, then inspect the normalization path in qlib/scripts/data_collector/base.py. Reproduce the update using the documented qlib.run.get_data command and determine whether current Yahoo data can complete normalization without the reported datetime TypeError.