microsoft / microsoft/qlib

执行yahoo normalize data 发生 ValueError: unconverted data remains when parsing with format "%Y-%m-%d" 错误

Open
#2,014 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

🐛 Bug Description

To Reproduce

Steps to reproduce the behavior:

1.python scripts/data_collector/yahoo/collector.py normalize_data --source_dir ~/.qlib/stock_data/source/cn_data --normalize_dir ~/.qlib/stock_data/normalize/cn_data --region CN --interval 1d
2.我已经下载了cn_data数据,normalize 过程中,在第341 sh600489.csv 发生错误

Expected Behavior

Screenshot

File "E:\Data\PROGRAM\Python\source\qlib\scripts\data_collector\yahoo\collector.py", line 395, in normalize_yahoo
df.index = pd.to_datetime(df.index)
...

ValueError: unconverted data remains when parsing with format "%Y-%m-%d": " 09:30:00+08:00", at position 1343. You might want to try:
- passing format if your strings have a consistent format;
- passing format='ISO8601' if your strings are all ISO8601 but not necessarily in exactly the same format;
- passing format='mixed', and the format will be inferred for each element individually. You might want to use dayfirst alongside this.

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

  • Qlib version:0.9.7
  • Python version: 3.12.9
  • OS (Windows, Linux, MacOS): windows

下载的原始数据
sh600468.csv
sh600469.csv

归一后的数据

sh600467.csv
sh600468.csv

  • Commit number (optional, please provide it if you are using the dev version):

Additional Notes

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/data_collector/yahoo/collector.py, especially normalize_yahoo around line 395, and reproduce the issue with the normalize_data command from the report. Inspect the attached CSV data around the failing row and how its timestamp format reaches pandas. Done means the supplied dataset completes normalization without the reported ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.