Ticker name "NA" makes the exists_qlib_data function report errors.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 48.7k
- Forks
- 7.7k
- PR merge metrics
- No merged PRs in 30d
Description
🐛 Bug Description
The ticker name "NA" in the "all.txt" under /instruments makes the exists_qlib_data function fail due to the string "NA" being wrongly converted to the float "nan" but not a string.
To Reproduce
Steps to reproduce the behavior:
- Save the attached all.txt under the
~/.qlib/qlib_data/us_data/instruments. - Run the following code:
provider_uri = "~/.qlib/qlib_data/us_data_new" # target_dir
if not exists_qlib_data(provider_uri):
print(f"Qlib data is not found in {provider_uri}")
sys.path.append(str(scripts_dir))
from get_data import GetData
GetData().qlib_data(target_dir=provider_uri, region=REG_US)
Expected Behavior
The code should run without errors.
Screenshot
Environment
Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.
- Qlib version: 0.93
- Python version: 3.8.10
- OS (
Windows,Linux,MacOS): Windows - Commit number (optional, please provide it if you are using the dev version):
Additional Notes
- The bug is caused by the wrong usage of pandas.read_csv in the following line of
exists_qlib_dataunderqlib\utils\__init__.py. Refer to the page for more details.
miss_code = set(pd.read_csv(_instrument, sep="\t", header=None).loc[:, 0].apply(str.lower)) - set(code_names)
- The cause of the bug can be further verified by the following code:
temp = pd.read_csv("all.txt", sep="\t", header=None).loc[:, 0]
non_string_values = [i for i in temp if not isinstance(i, str)]
print(non_string_values)
[nan]
- The bug can be easily fixed by adding
keep_default_na=False
temp = pd.read_csv("all.txt", sep="\t", header=None, keep_default_na=False).loc[:, 0]
non_string_values = [i for i in temp if not isinstance(i, str)]
print(non_string_values)
[]
- I can help with the fix, just want to ask what tests I need to run to make sure whether the fix would cause any other issues.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with exists_qlib_data in qlib/utils/init.py and reproduce the failure using the attached all.txt under the specified instruments directory. Verify that the ticker name "NA" remains a string and that the data check runs without errors, while checking the relevant pandas.read_csv behavior for regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100