microsoft / microsoft/qlib

Improve PIT performance

Open
#1,671 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

🌟 Feature Description

Current PIT implementation has a lot of performance trap and should be fixed.

Motivation

PIT feature is about 100 times slower than normal feature which is ridiculous. Financial PIT data usually have four points per year so it should be 50 faster than normal feature.

During review PIT code, I found following problems:

  1. PIT part

https://github.com/microsoft/qlib/blob/ecbeeafdc141ed89d5daf37ddfa20717190dfdb1/qlib/data/pit.py#L23-L48

In line 28, we loop every step in time series, and in each _load_feature function in line 39, we read the whole data file and index file.

Which makes about 1000 times slower for 1000 trade day. data file and index file should be read only once for one feature.

  1. LocalPITProvider part

_load_feature is actually implemented here.

https://github.com/microsoft/qlib/blob/ecbeeafdc141ed89d5daf37ddfa20717190dfdb1/qlib/data/data.py#L787-L794

Here, we read the whole data file but we pass data_path to nested function instead of data object!

https://github.com/microsoft/qlib/blob/ecbeeafdc141ed89d5daf37ddfa20717190dfdb1/qlib/data/data.py#L813-L817

This will cause another 2 times slower.

  1. read_period_data part

Line 150, read file in python loop:

https://github.com/microsoft/qlib/blob/ecbeeafdc141ed89d5daf37ddfa20717190dfdb1/qlib/utils/__init__.py#L147-L156

OK, this may be acceptable in C but not in python. Python loop is very slow and even in C, deal with file content without stream buffer is not recommend.

Alternatives

Use current slow implementation.

Additional Notes

I'll try to re-implement PIT workflow.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the referenced PIT implementation in qlib/data/pit.py, then inspect LocalPITProvider in qlib/data/data.py and read_period_data in qlib/utils/init.py. Benchmark the current PIT workflow against normal features, then verify that repeated file reads and the Python file-processing loop are removed while PIT results remain correct and performance improves substantially.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.