statsmodels / statsmodels/statsmodels

ShortPanel, too few individuals, singular

Open
#284 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

comp-regression corner-case type-enh
Dominant language
Python
Stars
11.6k
Forks
3.6k
Avg merge
7h 37m
Merged PRs (30d)
96

Description

The Grunfeld example has fewer firms than time periods

estimating an unrestricted covariance matrix results in a reduced rank matrix and causes a failure of the Cholesky decomposition.

Stata produces numbers, but prints a warning "not converged"

>>> mod = ShortPanelGLS2(res.model.endog, res.model.exog, firm_id)
>>> res_p = mod.fit()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef_new\statsmodels\sandbox\panel\panel_short.py", line 120, in fit
    self.cholsigmainv_i = np.linalg.cholesky(np.linalg.pinv(sigma_i)).T
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 531, in cholesky
    Cholesky decomposition cannot be computed'
numpy.linalg.linalg.LinAlgError: Matrix is not positive definite -         Cholesky decomposition cannot be computed

>>> sigma_i = mod.get_within_cov(mod.res_pooled.resid)
>>> np.linalg.eigvals(sigma_i)
array([ 118599.13113534,   28749.15411872,   10884.5047154 ,
          1567.26735086,     455.28185451,     283.0793974 ,
           118.27313134,      47.77001344,      44.56802788,
            34.72886823,       5.18697788,       0.        ,
            -0.        ,       0.        ,       0.        ,
            -0.        ,      -0.        ,      -0.        ,
            -0.        ,       0.        ])

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with statsmodels/sandbox/panel/panel_short.py at the fit path for ShortPanelGLS2 and reproduce the Grunfeld example with fewer firms than time periods. Investigate the reduced-rank covariance and Cholesky failure; done should mean this case no longer fails unexpectedly and its behavior is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.