numpy / numpy/numpy

polyfit LinAlgError: SVD did not converge in Linear Least Squares

Open
#17,593 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

After running my code on a new computer I had an issue with polyfit, which raises an error that should not happen.
Initially I ran my code on a Mac, with numpy 1.15.1.
The new computer is a laptop running on windows 10.
I tried several versions of numpy (1.19.1, 1.18.5, 1.17.5) but it didn't help and I could not dowlod 1.16.6 or 1.15.1 (because they are too old I guess)
My initial code was ;

import numpy as np
...
p0=np.polyfit(omega[nInit:nFin]-CentOmega,phi1[nInit:nFin],2,w=Spectrum[nInit:nFin])

omega, phi1 and Spectrum are numpy arrays (shape (4096,) ), nInit and nFin are within range and CentOmega is a scalar. I checked that there is no NaN values or infinites and that spectrum is positive.

Error message:

LinAlgError Traceback (most recent call last)
in calcCS2()
285 #print(np.min(phi1[nInit:nFin]))
286 #Here we remove the GDD to find the pulse shape with only higher order disp
--> 287 p0=np.polyfit(omega[nInit:nFin]-CentOmega,phi1[nInit:nFin],2,w=Spectrum[nInit:nFin]) #fitting
288 while True:
289 try:

<array_function internals> in polyfit(*args, **kwargs)

c:\users\pierre-marc\appdata\local\programs\python\python38\lib\site-packages\numpy\lib\polynomial.py in polyfit(x, y, deg, rcond, full, w, cov)
629 scale = NX.sqrt((lhs*lhs).sum(axis=0))
630 lhs /= scale
--> 631 c, resids, rank, s = lstsq(lhs, rhs, rcond)
632 c = (c.T/scale).T # broadcast scale coefficients
633

<array_function internals> in lstsq(*args, **kwargs)

c:\users\pierre-marc\appdata\local\programs\python\python38\lib\site-packages\numpy\linalg\linalg.py in lstsq(a, b, rcond)
2266 # lapack can't handle n_rhs = 0 - so allocate the array one larger in that axis
2267 b = zeros(b.shape[:-2] + (m, n_rhs + 1), dtype=b.dtype)
-> 2268 x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
2269 if m == 0:
2270 x[...] = 0

c:\users\pierre-marc\appdata\local\programs\python\python38\lib\site-packages\numpy\linalg\linalg.py in _raise_linalgerror_lstsq(err, flag)
107
108 def _raise_linalgerror_lstsq(err, flag):
--> 109 raise LinAlgError("SVD did not converge in Linear Least Squares")
110
111 def get_linalg_error_extobj(callback):

LinAlgError: SVD did not converge in Linear Least Squares

NumPy 1.17.5 / Python 3.8.6

I found a way to avoid this error by doing the following:

while True: 
        try: 
            p0=np.polyfit(omega[nInit:nFin]-CentOmega,phi1[nInit:nFin],2,w=Spectrum[nInit:nFin]) #fitting 
            break 
        except: 
            continue

I saw this on stackoverflow for people who had the same issue. They noticed running the same code a second time works.
This is really a stupid code, and should not work (but it does). All it says is keep doing the same thing until you don't get an error
I don't understand why successive calls to polyfit with the same parameters would give a different result.

I didn't find this bug in your list of known issues but maybe someone else reported it?

I hope you can fix this in later versions.

Best regards,

Pierre-Marc Dansette

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the np.polyfit call and inspect the traceback locations in numpy/lib/polynomial.py and numpy/linalg/linalg.py. First obtain a minimal reproducible set of omega, phi1, Spectrum, nInit, nFin, and CentOmega that triggers the Windows failure, then compare repeated calls across the reported NumPy versions. Done means the cause is isolated or the failure is reproducibly characterized.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.