microsoft / microsoft/qlib

Qlib Features Calculation error

Open
#1,874 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

Hello, I am pretty new to qlib scripting so please bare with me. I am currently running into issues with feature calculations and how that process exactly works. for some background information, I am trying to get it to process my own data from a csv file, it is stock market data. And it has the correct formation with datetime, symbol, open, high, low, close, volume. And I want to make it so that my script can process this data and calculate all 158 features for the data. Please let me know if that is possible, and if it is, I would appreciate some pseudocode for it. Thank you so much.
I have provided some code below for more context:

import qlib
from qlib.contrib.data.handler import Alpha158
import pandas as pd
import os
import logging

Enable detailed logging for debugging

logging.basicConfig(level=logging.DEBUG)

def initialize_qlib(data_dir):
logging.info(f"Initializing Qlib with directory: {data_dir}")
qlib.init(provider_uri=data_dir)
logging.info("Qlib initialized successfully.")

def prepare_qlib_data(csv_path, output_dir):
logging.info(f"Preparing data from CSV: {csv_path}")
df = pd.read_csv(csv_path)
logging.debug(f"Original DataFrame:\n{df.head()}")

# Ensure datetime parsing
if 'datetime' in df.columns:
    df['datetime'] = pd.to_datetime(df['datetime'], errors='coerce')
elif 'date' in df.columns:
    df['datetime'] = pd.to_datetime(df['date'], errors='coerce')
else:
    raise ValueError("CSV file must have a 'datetime' or 'date' column.")

# Drop invalid rows
df.dropna(subset=['datetime'], inplace=True)
df.set_index(['datetime', 'symbol'], inplace=True)

# Save to parquet
os.makedirs(output_dir, exist_ok=True)
output_file = os.path.join(output_dir, 'data.parquet')
df.to_parquet(output_file)
logging.info(f"Data prepared and saved to: {output_file}")

def setup_alpha158_handler(data_dir):
logging.info("Setting up Alpha158 data handler.")

# Define configuration for Alpha158
data_handler_config = {
    "start_time": "2013-12-30",
    "end_time": "2024-12-24",
    "fit_start_time": "2015-01-01",
    "fit_end_time": "2017-01-01",
    "instruments": "AAPL",
}

# Initialize the handler
handler = Alpha158(**data_handler_config)
logging.info("Alpha158 handler initialized successfully.")
return handler

def calculate_features(handler, instrument, freq):
try:
logging.info(f"Calculating features for {instrument} at {freq} frequency.")

    # Debug available columns
    logging.debug(f"Columns in data: {handler.get_cols()}")
    
    # Fetch labels
    labels = handler.fetch(col_set="label")
    logging.debug(f"Labels:\n{labels.head()}")
    
    # Fetch features
    features = handler.fetch(col_set="feature")
    logging.debug(f"Features:\n{features.head()}")
    
    print("Features and labels calculated successfully.")
    return features, labels

except Exception as e:
    logging.error(f"Error calculating features for {instrument}: {e}")
    raise

if name == "main":
csv_path = "C:/Users/14168/.qlib/qlib_data/data/historical (3).csv" # Input CSV
data_dir = "C:/Users/14168/.qlib/qlib_data/" # Qlib Data Directory

prepare_qlib_data(csv_path, data_dir)
initialize_qlib(data_dir)

# Calculate and display features
calculate_features('AAPL', freq='day')

try:
    # Prepare data
    prepare_qlib_data(csv_path, data_dir)
    
    # Initialize Qlib
    initialize_qlib(data_dir)
    
    # Setup Alpha158
    handler = setup_alpha158_handler(data_dir)
    
    # Calculate and print features
    features, labels = calculate_features(handler, instrument='AAPL', freq='day')
except Exception as e:
    logging.error(f"An error occurred: {e}")

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue centers on the shown prepare_qlib_data, setup_alpha158_handler, and calculate_features functions using Alpha158. Start by checking Qlib's documented data format and Alpha158 usage, then reproduce the reported CSV-to-feature flow. Done would require a clear, verified example explaining whether this input format can produce the requested features and correcting the usage guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.