Improve PIT performance
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 48.7k
- Forks
- 7.7k
- PR merge metrics
- No merged PRs in 30d
Description
🌟 Feature Description
Current PIT implementation has a lot of performance trap and should be fixed.
Motivation
PIT feature is about 100 times slower than normal feature which is ridiculous. Financial PIT data usually have four points per year so it should be 50 faster than normal feature.
During review PIT code, I found following problems:
- PIT part
In line 28, we loop every step in time series, and in each _load_feature function in line 39, we read the whole data file and index file.
Which makes about 1000 times slower for 1000 trade day. data file and index file should be read only once for one feature.
- LocalPITProvider part
_load_feature is actually implemented here.
Here, we read the whole data file but we pass data_path to nested function instead of data object!
This will cause another 2 times slower.
- read_period_data part
Line 150, read file in python loop:
OK, this may be acceptable in C but not in python. Python loop is very slow and even in C, deal with file content without stream buffer is not recommend.
Alternatives
Use current slow implementation.
Additional Notes
I'll try to re-implement PIT workflow.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the referenced PIT implementation in qlib/data/pit.py, then inspect LocalPITProvider in qlib/data/data.py and read_period_data in qlib/utils/init.py. Benchmark the current PIT workflow against normal features, then verify that repeated file reads and the Python file-processing loop are removed while PIT results remain correct and performance improves substantially.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100