statsmodels / statsmodels/statsmodels
ShortPanel, too few individuals, singular
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.6k
- Forks
- 3.6k
- Avg merge
- 7h 37m
- Merged PRs (30d)
- 96
Description
The Grunfeld example has fewer firms than time periods
estimating an unrestricted covariance matrix results in a reduced rank matrix and causes a failure of the Cholesky decomposition.
Stata produces numbers, but prints a warning "not converged"
>>> mod = ShortPanelGLS2(res.model.endog, res.model.exog, firm_id)
>>> res_p = mod.fit()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef_new\statsmodels\sandbox\panel\panel_short.py", line 120, in fit
self.cholsigmainv_i = np.linalg.cholesky(np.linalg.pinv(sigma_i)).T
File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 531, in cholesky
Cholesky decomposition cannot be computed'
numpy.linalg.linalg.LinAlgError: Matrix is not positive definite - Cholesky decomposition cannot be computed
>>> sigma_i = mod.get_within_cov(mod.res_pooled.resid)
>>> np.linalg.eigvals(sigma_i)
array([ 118599.13113534, 28749.15411872, 10884.5047154 ,
1567.26735086, 455.28185451, 283.0793974 ,
118.27313134, 47.77001344, 44.56802788,
34.72886823, 5.18697788, 0. ,
-0. , 0. , 0. ,
-0. , -0. , -0. ,
-0. , 0. ])
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with statsmodels/sandbox/panel/panel_short.py at the fit path for ShortPanelGLS2 and reproduce the Grunfeld example with fewer firms than time periods. Investigate the reduced-rank covariance and Cholesky failure; done should mean this case no longer fails unexpectedly and its behavior is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100