microsoft / microsoft/qlib

FileFeatureStorage lowercases the instrument directory name when building feature paths

Open
#2,053 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

FileFeatureStorage lowercases the instrument directory name when building feature paths (instrument.lower()/field.lower().freq.bin). On case‑sensitive filesystems (e.g., Linux) where the vendor data is stored with uppercase instrument folders (data/xsto/features/ABB-U/...), the loader looks for abb-u/…, can’t find the file, and every expression resolves to an empty dataframe even though the data exists.

To Reproduce

Steps to reproduce the behavior:

Store local feature data under uppercase instrument directories (e.g., data/xsto/features/ABB-U/close.day.bin).
Initialize qlib with that provider URI.
Request any feature for that instrument, e.g. D.features(["ABB-U"], ["$close"], "2000-01-01", "2025-12-31"). The result is an empty dataframe and the feature storage backend reports the file is missing.

Expected Behavior

FileFeatureStorage should respect the actual on-disk casing instrument list (or probe common case variants) so that features load correctly regardless of filesystem case sensitivity.

Screenshot

N/A – the console logs show statements such as [FileFeatureStorage.getitem] uri=…/features/abb-u/close.day.bin missing for request=… even though …/features/ABB-U/close.day.bin exists.

Environment

Version: 0.9.7
Python version: 3.12 (virtual env)
OS: Linux (Ubuntu on AMD cloud server, case-sensitive filesystem)
Commit number: N/A (using pip wheel)
Additional Notes
We patched FileFeatureStorage locally to keep multiple filename candidates (instrument.lower(), original case, uppercase) and select the first existing path before reading. After this change ABB-U and other uppercase symbols load correctly. Happy to submit a PR if this approach sounds acceptable.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate FileFeatureStorage and trace how instrument and field names become feature paths. Reproduce with data/xsto/features/ABB-U/close.day.bin and a D.features request for ABB-U/$close; the fix is complete when the existing uppercase file loads on a case-sensitive filesystem instead of returning an empty dataframe.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.