microsoft / microsoft/qlib

Questions about 'get_feature_config' function in class HighFreqGeneralHandler

Open
#1,786 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

When I run the follwing code
python scripts/gen_pickle_data_hsm.py -c scripts/pickle_data_config.yml
, I print out the generated results of self.get_feature_config() in class HighFreqGeneralHandler. The results are as follows:
['Cut(FFillNan(If(IsNull($open), $close, $open)/DayLast(Ref(FFillNan($close), 480))), 480, None)',
'Cut(FFillNan(If(IsNull($high), $close, $high)/DayLast(Ref(FFillNan($close), 480))), 480, None)',
'Cut(FFillNan(If(IsNull($low), $close, $low)/DayLast(Ref(FFillNan($close), 480))), 480, None)',
'Cut(FFillNan(If(IsNull($close), $close, $close)/DayLast(Ref(FFillNan($close), 480))), 480, None)',
'Cut(FFillNan(Ref(If(IsNull($open), $close, $open), 240)/DayLast(Ref(FFillNan($close), 240))), 480, None)',
'Cut(FFillNan(Ref(If(IsNull($high), $close, $high), 240)/DayLast(Ref(FFillNan($close), 240))), 480, None)',
'Cut(FFillNan(Ref(If(IsNull($low), $close, $low), 240)/DayLast(Ref(FFillNan($close), 240))), 480, None)',
'Cut(FFillNan(Ref(If(IsNull($close), $close, $close), 240)/DayLast(Ref(FFillNan($close), 240))), 480, None)',
'Cut(If(IsNull($volume/Ref(DayLast(Mean($volume, 7200)), 240)), 0, $volume/Ref(DayLast(Mean($volume, 7200)), 240)), 480, None)',
'Cut(If(IsNull(Ref($volume, 240)/Ref(DayLast(Mean($volume, 7200)), 240)), 0, Ref($volume, 240)/Ref(DayLast(Mean($volume, 7200)), 240)), 480, None)'
], ['$open', '$high', '$low', '$close', '$open_1', '$high_1', '$low_1', '$close_1', '$volume', '$volume_1']

I has 2 questions here:
First, I believe the variable 'day length' in class HighFreqGeneralHandler should be divided by freq. In my case, the freq is 5 min, so a day only has a length of 48. The Ref($close, 480) here just indicates the close price 10 days ago.
Second, the actual usage of function 'get_normalized_price_feature' is inconsistent with the annotations here. However, it seems that the open,close,high,low of today will be divided by the close price 2 days ago, and the those of yesterday will be divided by the close price of yesterday(and then Ref again? I don't know why). I 'm a little confused here and is looking forward for a more detailed explanation.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run python scripts/gen_pickle_data_hsm.py -c scripts/pickle_data_config.yml and inspect HighFreqGeneralHandler.get_feature_config and get_normalized_price_feature, comparing their annotations with the generated expressions. Trace how freq and day length are used in the normalization and Ref calls. Done means providing a confirmed explanation or identifying a reproducible implementation mismatch and its expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, fintech-quant
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.