microsoft / microsoft/qlib

Ticker name "NA" makes the exists_qlib_data function report errors.

Open
#1,720 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

🐛 Bug Description

The ticker name "NA" in the "all.txt" under /instruments makes the exists_qlib_data function fail due to the string "NA" being wrongly converted to the float "nan" but not a string.

To Reproduce

Steps to reproduce the behavior:

  1. Save the attached all.txt under the ~/.qlib/qlib_data/us_data/instruments.
  2. Run the following code:
provider_uri = "~/.qlib/qlib_data/us_data_new"  # target_dir
if not exists_qlib_data(provider_uri):
    print(f"Qlib data is not found in {provider_uri}")
    sys.path.append(str(scripts_dir))
    from get_data import GetData

    GetData().qlib_data(target_dir=provider_uri, region=REG_US)

Expected Behavior

The code should run without errors.

Screenshot

image

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

  • Qlib version: 0.93
  • Python version: 3.8.10
  • OS (Windows, Linux, MacOS): Windows
  • Commit number (optional, please provide it if you are using the dev version):

Additional Notes

  1. The bug is caused by the wrong usage of pandas.read_csv in the following line of exists_qlib_data under qlib\utils\__init__.py. Refer to the page for more details.
 miss_code = set(pd.read_csv(_instrument, sep="\t", header=None).loc[:, 0].apply(str.lower)) - set(code_names)
  1. The cause of the bug can be further verified by the following code:
temp = pd.read_csv("all.txt", sep="\t", header=None).loc[:, 0]
non_string_values = [i for i in temp if not isinstance(i, str)]
print(non_string_values)
[nan]
  1. The bug can be easily fixed by adding keep_default_na=False
temp = pd.read_csv("all.txt", sep="\t", header=None, keep_default_na=False).loc[:, 0]
non_string_values = [i for i in temp if not isinstance(i, str)]
print(non_string_values)
[]
  1. I can help with the fix, just want to ask what tests I need to run to make sure whether the fix would cause any other issues.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with exists_qlib_data in qlib/utils/init.py and reproduce the failure using the attached all.txt under the specified instruments directory. Verify that the ticker name "NA" remains a string and that the data check runs without errors, while checking the relevant pandas.read_csv behavior for regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.