Generating One Generalized Model for Multiple Datasets?

Open
#5,549 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
data

Research direction

The issue mentions VARMAX and concatenating multivariate time series, but names no repository files or tests. Start by reviewing the VARMAX entry point and the linked statsmodels discussion; a contribution would need a defined supported workflow and tests, which the issue does not specify.

Written by the indexing model from the issue text.

Description

I have multiple sets of multivariate time series which each represent health data for different people. The datasets share the same set of features, but contain varying numbers of observations. I'm assuming the data is stationary and the same underlying process governs each set of data. I want to make ONE generalized model for all the time series.

As of now, my approach has been modeled after this answer by Rob Hyndman. Essentially you concatenate all the multivariate time series together with a set of zeros between each dataset.

I fit a VARMAX model on to this large data set from a previous year and then use the method described by @ChadFulton here to make predictions on each individual dataset for the current year.

I see a couple of potential issues with the methodology I am currently using. The biggest is that even though the same process describes the interaction of each set of multivariate data. The data for each person is on different scales. I think that this would disrupt the fitting process of the model? Is there a way to fit a model on every dataset individually, essentially averaging the fits so that the variance in scale between datasets isn't an issue?

One way I have thought about solving this is to perform a standard scale of each individual dataset before concatenating them together. I'm not super well-versed in this field, but I think this would cause the data sets with larger values to lose some of their descriptory power. Is there a way to call loop a fit over many data sets and essentially average the fits based on the number of observations for each fit?

Dominant language
Python
Stars
11.6k
Forks
3.6k
Avg merge
7h 37m
Merged PRs (30d)
96

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from statsmodels/statsmodels

All issues in statsmodels/statsmodels

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.