microsoft / microsoft/qlib

Why did I get "killed" when running 1min of alpha360?

Open
#1,761 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
48.7k
Forks
7.7k
PR merge metrics
No merged PRs in 30d

Description

###################################
# train model
###################################
data_handler_config = {
    "start_time":"2020-09-14",
    "end_time":"2021-06-20",
    "fit_start_time":"2020-09-20",
    "fit_end_time":"2021-05-20",
    "freq":'1min',
    "instruments": market,
}

task = {
    "model": {
        "class": "LGBModel",
        "module_path": "qlib.contrib.model.gbdt",
        "kwargs": {
            "loss": "mse",
            "colsample_bytree": 0.8879,
            "learning_rate": 0.0421,
            "subsample": 0.8789,
            "lambda_l1": 205.6999,
            "lambda_l2": 580.9768,
            "max_depth": 8,
            "num_leaves": 210,
            "num_threads": 20,
        },
    },
    "dataset": {
        "class": "DatasetH",
        "module_path": "qlib.data.dataset",
        "kwargs": {
            "handler": {
                "class": "Alpha360",
                "module_path": "qlib.contrib.data.handler",
                "kwargs": data_handler_config,
            },
            "segments": {
    "train": ("2020-09-14", "2020-10-15"),
    "valid": ("2020-10-16", "2020-12-20"),
    "test": ("2020-12-14", "2021-05-20"),
            },
        },
    },
}


h = Alpha360(**data_handler_config)
alpha360_df_feature = h.fetch(col_set="feature")
alpha360_df_feature = alpha360_df_feature.loc[:,~(alpha360_df_feature == 0.0).all()]
alpha360_df_feature = alpha360_df_feature.loc[~(alpha360_df_feature.eq(0.0).all(1))]

print(alpha360_df_feature)

# model initiaiton
model = init_instance_by_config(task["model"])
dataset = init_instance_by_config(task["dataset"])

# start exp to train model
with R.start(experiment_name="train_model"):
    R.log_params(**flatten_dict(task))
    model.fit(dataset)
    R.save_objects(trained_model=model)
    rid = R.get_recorder().id


image

when I try tu run jupyter notebook, it always showed that python kernel has been restarted.

What's datetime range does it support? I tried one day range, it works fine from 2020-09-14 to 2020-09-15,
Most of feature in alpha360 are zero. this range may a little bit short.

My machine:
Debian 11
RAM 64G
4 processors

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the one-day and longer-range cases with Alpha360.fetch, then compare behavior around model.fit using the supplied LGBModel and DatasetH configuration. Inspect the notebook kernel or process memory while expanding the date range. Done means documenting the supported datetime range and identifying whether the restart is caused by resource use or a data issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
data, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.