[svd_intro] Some suggestions on code and style

Hi @jstac

I was reading through the svd_intro lecture with codex and found a few places that might worth an update. I’d be grateful if you could take a look when you have a chance.

Overview

• At line 21-25, change “four fundamental spaces” to “four fundamental subspaces,” which is the subsection heading used later. Remove the promise to cover overdetermined least-squares regression, which does not appear later.

• At line 25-29, introduce abbreviations consistently. PCA has already been defined before, whereas DMD is used before it is expanded. Suggested change:

  Like PCA, dynamic mode decomposition (DMD) can be thought of as …

• At line 18-29, typos: work-horse → workhorse, SVD's → SVDs, and “provide ways rapidly to compute” → “provide ways to rapidly compute.”

The setting

• At line 37-57, use “observations” as the generic term for columns, since the columns can represent either people or time periods, and describe the rows consistently as variables or attributes.

• At line 46-57, replace <<, < <, >>, and > > with the mathematical relations $\ll$ and $\gg$.

• At line 51-59, use the established abbreviations SVD, PCA, and DMD. Also clarify that DMD is constructed in the sequel rather than later in this lecture. Suggested update:

  In the sequel, we’ll use an SVD to construct a DMD.

• At line 59, add the missing period.

Singular value decomposition

• At line 94-104, update the geometric interpretation. The orthogonal factors can rotate or reflect vectors while preserving lengths and angles; a general diagonal $\Sigma$ rescales different coordinate directions and does not generally preserve angles.

• At line 80-91, make the complex-valued case internally consistent. For complex matrices, the factors are unitary and it is currently called orthogonal which is the same for real matrix, and conjugate transpose should use a different notation from ordinary transpose. In addition, $U_{ij}^{\top}$ does not represent complex conjugation of a scalar entry. We could update this part by introducing adjoint notation, such as $(U^*){ij}=\overline{U{ji}}$.

• At line 86, we should also mention that eigenvalues can be zero.

• At line 115, we didn't discuss DMD in this lecture so replace “later in this lecture” with a reference to the sequel, where DMD is actually treated.

• At line 94,125, change “reshape in vectors” to “reshape vectors” or “act on vectors,” and add the comma in “Before we study a reduced SVD, we’ll …”

Four fundamental subspaces

• At line 159-163, the current expression defines $U_R$ twice and ends with $u_n$. We could update the second definition of $U_R$ to

$$
V_R=\begin{bmatrix}v_{p+1}&\cdots&v_n\end{bmatrix}.
$$

• At line 184-191,221-228, the images are $\sigma_i u_i$ and $\sigma_i v_i$. They are mutually orthogonal, with norms $\sigma_i$ but currently, we are saying they are orthronormal.

• At line 299-321, NumPy returns U, s, Vh, where Vh is $V^\top$. With basis vectors stored as columns, the four subspaces should be formed as follows (currently, we are taking Vh as V)

  U, s, Vh = np.linalg.svd(A, full_matrices=True)
  
  col_space = U[:, :rank]
  left_null_space = U[:, rank:]
  row_space = Vh[:rank, :].T
  null_space = Vh[rank:, :].T
  

  So we could update the later print references and labels to use s and Vh as part of the same edit. The current symmetric example masks the row-space/column-space swap, but the computed null-space basis is not a null-space basis.

• At line 135-274, add commas after the notation clauses at lines 135 and 137; change “consist” to “consists” at line 139; put $X$ in math delimiters at line 150; change i.e, to i.e., at line 265; and add “of” in “the orthogonal complement of ${\mathcal N}(X)$” at line 274.

Eckart–Young theorem

• At line 343-347, order the positive singular values as $\sigma_1\geq\cdots\geq\sigma_p>0$ so that our claim the first $r$ terms would produce the best approximation. Also we should use lowercase notation here keeps the formula consistent with the notation established earlier in the lecture. Also we should say "a best rank-$r$ approximation" instead of "the best rank-$r$ approximation" as the latter implies uniqueness.

• At line 349, replace the rank and dimension description. $\hat X_r$ remains $m\times n$ and has rank $r<p$; $X$ itself need not be rank deficient.

• At line 353-357, retain the established abbreviations and distinguish the locations of the applications: PCA is discussed below, while DMD is treated in the sequel.

• At line 328-341, write $\ell^2$ rather than $l^2$, change “denotes a norm of a matrix $X$” to “denotes a matrix norm,” and repair the affected grammar.

Full and reduced SVDs

• At line 363-370,417-421,474-475, clarify that the lecture’s reduced SVD is not always the same as NumPy’s full_matrices=False output. The lecture retains only the $p=rank(X)$ positive singular values, giving factors $U_p$, $\Sigma_p$, and $V_p^\top$ with shapes $(m,p)$, $(p,p)$, and $(p,n)$. NumPy instead retains $k=\min(m,n)$ singular-value positions and returns arrays with shapes $(m,k)$, $(k,)$, and $(k,n)$. When $p<k$, its output includes $k-p$ zero-singular-value directions. The two versions retain the same factors only when $p=k$; otherwise, obtain the lecture’s reduced factors by slicing:

  U_p = U[:, :p]
  Sigma_p = np.diag(s[:p])
  Vh_p = Vh[:p, :]
  

• At line 381-400, add the missing rank conditions. The reduced factors always satisfy

$$
U^\top U=V^\top V=I_p.
$$

The matrices $UU^\top$ and $VV^\top$ are orthogonal projectors. In the tall case, $VV^\top=I_n$ only when $p=n$; in the wide case, $UU^\top=I_m$ only when $p=m$.

• At line 427-506, use NumPy’s actual return notation U, s, Vh, making clear that s is a vector and Vh is $V^\top$. For the thin output, reconstruct with

  U_hat, s_hat, Vh_hat = np.linalg.svd(X, full_matrices=False)
  X_hat = U_hat @ np.diag(s_hat) @ Vh_hat
  

  For the full output, first construct the rectangular $\Sigma$ before multiplying U @ Sigma @ Vh.

• At line 403,425,479, replace references to DMD “below” with “in the sequel” and use the established abbreviation DMD.

• At line 359-506, write $\ll$ and $\gg$ for the comparison relations, change “exercise” to “example,” use SVDs rather than SVD's, and add the missing periods.

Polar decomposition

• At line 519-529, replace the claim that $Q=UV^\top$ is orthogonal with: “$Q$ is an $m\times n$ matrix that preserves lengths on the row space of $X$.”

• At line 521-528, describe $S=U\Sigma U^\top$ as symmetric positive semidefinite and remove bold from “symmetric.” Positive semidefiniteness is required because the eigenvalues of $S$ are the nonnegative singular values.

• At line 531-535, say that $U$ and $V$ are matrices with orthonormal columns, and add the missing word “matrix” after the description of $V$.

• At line 511-535, begin “A reduced SVD …” and add the missing period.

Application: PCA

• At line 537-553, use “observations” as the generic description of columns, remove “to be discussed later,” write $n\gg m$ and $m\gg n$, and avoid the collision between notation for observations and variables. For example, retain $X_i$ for variables and write

$$
X=[x_1\mid\cdots\mid x_n],
\qquad
x_j=[x_{1j},\ldots,x_{mj}]^\top
$$

for observation columns.

• At line 583-606, replace “positive definite” with “symmetric positive semidefinite.” The matrix

$$
C=\frac{1}{n}BB^\top
$$

is positive definite only when $B$ has full row rank.

• At line 591-609, order entire singular triplets consistently: if the singular values are reordered, reorder the corresponding columns of both $U$ and $V$ with them.

• At line 622-629, correct the PCA score formula for variables in rows and observations in columns:

$$
T=U^\top B=\Sigma V^\top.
$$

For $r$ retained components, use $T_r=U_r^\top B=\Sigma_rV_r^\top$. The current $BV=U\Sigma$ identity is true but is not the score matrix in this orientation.

• At line 612-620, call the displayed explained-variance ratio a “proportion” unless it is multiplied by 100, and say “total variance.”

Relationship of PCA to SVD

• At line 640-659, use the established lowercase singular-vector notation and specify the rank-one range $j=1,\ldots,p$:

$$
X=\sum_{j=1}^{p}\sigma_j u_jv_j^\top.
$$

• At line 665, the lecture identifies ${V_{kj}}_{j=1}^n$ as the series for component $k$. This reverses the indices of $V$, omits the singular-value scaling, and lets $k$ run to $n$ rather than to the rank $p$. Since the score matrix is $T=U^\top X=\Sigma V^\top$, the $k$th component-score vector is $t_k^\top=\sigma_kv_k^\top$, with series

$$
{\sigma_kV_{jk}}_{j=1}^{n},
\qquad k=1,\ldots,p.
$$

• At line 667-668, correct the loading-vector indices:

$$
u_k=\begin{bmatrix}U_{1k}&\cdots&U_{mk}\end{bmatrix}^\top,
\qquad k=1,\ldots,p.
$$

• At line 670, state that component $k$ has variance $\sigma_k^2/n$ under the lecture’s covariance convention, and that its explained-variance share is

$$
\frac{\sigma_k^2}{\sum_{j=1}^{p}\sigma_j^2},
$$

rather than being proportional to $\sigma_k$.

• At line 640-668, use horizontal \cdots and vertical \vdots, keep “For each” outside math delimiters, and repair the affected punctuation.

PCA with eigenvalues and eigenvectors

• At line 674-686, preserve $\Omega=XX^\top$ but call it the centered-data cross-product or scatter matrix, not a sample covariance matrix. This keeps the subsequent scaling consistent without changing the derivation.

• At line 688-709, state that $P$ is orthogonal and that its columns are eigenvectors, so $P^{-1}=P^\top$ and $\epsilon=P^\top X$.

• At line 724-730, correct the data-matrix dimensions. Since there are $n$ observation columns, write $X=[x_1\mid\cdots\mid x_n]$; the expansion $X=\sum_{j=1}^{m}P_j\epsilon_j$ still has $m$ component terms.

• At line 733-750, replace the scalar-like identity with

$$
\epsilon_j\epsilon_j^\top=\lVert\epsilon_j\rVert^2=\lambda_j=\sigma_j^2,
$$

normalize only for $j=1,\ldots,p=rank(X)$, change the first $\tilde\epsilon_2$ at line 743 to $\tilde\epsilon_1$, and use $p$ rather than $m$ or the undefined $r$ as the upper limit of the nonzero expansion.

• At line 750-757, use lowercase $u_j,v_j$ and a common index, and distinguish the score from its normalized version:

$$
\epsilon_j=\sigma_jv_j^\top,
\qquad
\tilde\epsilon_j=v_j^\top.
$$

• At line 759-764, qualify the ambiguity discussion. Ordering and sign conventions can align distinct singular/eigenvector pairs, but repeated eigenvalues still allow rotations within an eigenspace. A sign convention should use a guaranteed nonzero entry, such as the largest-magnitude entry of each vector, rather than requiring positive diagonal entries.

• At line 674-696, use eigendecomposition rather than “eigen decomposition,” change “assure” to “ensure,” and add the missing article before $m\times m$.

Connections

• At line 770-821, specify that the displayed factorization is a full SVD and qualify $P=U$: after consistent ordering and sign choices, $P$ can be chosen to equal $U$, but repeated eigenspaces still permit rotations.

• At line 804-808, call $XX^\top$ the cross-product or scatter matrix rather than the sample covariance matrix, consistently with the preceding section.

• At line 841-855, state in the surrounding prose and the DecomAnalysis docstring that X must already be centered. The class computes self.Ω = X @ X.T without subtracting the variable means. For uncentered data,

$$
XX^\top=BB^\top+n\bar{x}\bar{x}^\top,
$$

so the resulting principal directions can reflect the data’s distance from the origin rather than variation around the sample mean.

• At line 851-934, repair component-count handling. Default r_component to the matrix rank rather than $m$, test explicitly with is not None, validate the requested count, and apply the same validation in fit(). The current default makes the SVD reconstruction fail for tall matrices. Also make the hidden call-order requirement explicit by either having fit() compute the decompositions or raising a clear error unless pca() and svd() have already run.

• At line 936-941, make diag_sign robust. Since np.sign(0) is zero, using diagonal entries can zero an entire column. Base each sign on a guaranteed nonzero entry, such as the largest-magnitude entry in that column.

• At line 961-981, plot the loading vectors as columns rather than plotting P.T and U.T, and make the two score plots use the same valid component range and scaling. For example, in the loading figure use

  q = da.r
  fig, axs = plt.subplots(1, 2)
  axs[0].plot(da.P[:, :q])
  axs[1].plot(da.U[:, :q])
  

  and in the separate score figure use

  fig, axs = plt.subplots(1, 2)
  axs[0].plot(da.ε[:q, :].T)
  axs[1].plot(da.VT[:q, :].T * da.σ[:q])
  

  This also avoids the shape mismatch in the current np.sqrt(da.λ) expression when da.r < m.

• At line 841-982, add a short execution example after the definitions of DecomAnalysis and compare_pca_svd. The current cells only define the class and function, so the plt.show() calls inside compare_pca_svd never execute and the lecture displays no numerical or graphical PCA–SVD comparison. Once the class and plotting corrections in this section are made, instantiate DecomAnalysis with centered data and call the comparison function, for example:

  X_centered = X - X.mean(axis=1, keepdims=True)
  da = DecomAnalysis(X_centered)
  compare_pca_svd(da)
  

• At line 965-980, replace the one-letter and dimension-only axis labels with descriptive labels for the plotted loadings and component scores.

• At line 802-895, add the missing period at line 802, use eigendecomposition, repair “for a given a,” and change the comment at line 895 to “sort by singular values.”

Exercises

• At line 989, make the regression orientation explicit or use a different symbol for the design matrix. Earlier $X$ stores observations in columns, whereas the OLS formula here conventionally uses observations in rows.

• At line 991-993, replace the determinant-based explanation. $X^\top X$ is singular when $X$ lacks full column rank, while a small determinant is scale dependent and does not by itself characterize ill-conditioning. Also avoid implying that an exact pseudoinverse cures full-rank ill-conditioning; that would require regularization or truncation.

• At line 993, describe the Moore–Penrose pseudoinverse as an exact generalized inverse, not a “full rank approximation.” It has the same rank $p$ as $X$.

• At line 995-1013, align the prompt and solution. The solution constructs the exact pseudoinverse and does not use the Eckart–Young theorem. A minimal prompt would be: “Using the SVD, construct the pseudoinverse $X^+$ and use it to compute $\hat\beta$.” If Eckart–Young is retained, the solution would instead need to introduce a truncated, regularized pseudoinverse.

• At line 1015-1031, state that $\Sigma^+$ is $n\times m$, with reciprocals of the first $p$ positive singular values, and that $\hat\beta=X^+y$ is the minimum-norm least-squares solution. Replace “inverting $X$” with language about forming its pseudoinverse.

• At line 989-1041, correct colinearity, under-determined, the duplicated not, the missing “of,” and SVD's; hyphenate “short-and-fat” when it is used attributively; add the missing comma; and repair the sentence fragments.

Style

• [qe-writing-001] At line 557,595, split the multi-sentence paragraphs into shorter paragraphs.

• [qe-writing-002] At line 557,563,576,595-620, there are some sentence fragments that need to be stated in whole, clarify the PCA description, state that covariance measures how variables vary together, and use a consistent narrative voice; at line 635, tighten the opening sentence; and at line 759, repair the malformed sentence and repetitive prose.

• [qe-writing-004] At line 956, change the printed label to “Eigenvalues and squared singular values”; and at line 989,1041, lowercase “ordinary least squares” and “vector autoregressions,” or use the established abbreviations OLS and VARs after expansion.

• [qe-writing-005] At line 23-25, remove bold from the three roadmap items while retaining bold on the first definition of singular value decomposition at line 18; at line 51,59, remove bold from the repeated mentions of “singular value decomposition” and “dynamic mode decomposition”; at line 96-104, remove bold used for ordinary emphasis on “rotates,” “angles between vectors,” “lengths of vectors,” “rescales,” “rotating,” and “rescaling”; at line 117-125, make “Road ahead” an H3 heading and remove repeated bold from “full” and “reduced” after their defining occurrences; at line 363-479, replace the bolded **Properties:**, which is being used as a heading rather than a definition, with an unnumbered rubric; at line 511-535, reserve bold for the first definition of polar decomposition and remove it from repeated or ordinary terms; at line 559,583,591,612,622, replace the five bold Step labels with unnumbered {rubric} directives and write their labels in sentence case—Step 1: center the data, Step 2: compute the covariance matrix, Step 3: decompose the covariance matrix and order the singular values, Step 4: select singular values and optionally truncate the rest, and Step 5: create the score matrix; at line 539-622, remove repeated bold from “short,” “tall,” and other previously introduced or ordinary terms; at line 665-670, retain bold for the first definitions of principal component and loadings, but remove the repeated bold at line 670; at line 680, change “pre-processing” to preprocessing and remove its bold; and at line 989, remove repeated bold from “short and fat.”

• [qe-writing-006] At line 31, change ## The Setting to ## The setting; at line 61, change ## Singular Value Decomposition to ## Singular value decomposition; at line 127, change ## Four Fundamental Subspaces to ## Four fundamental subspaces; at line 324, change the heading to ## Eckart-Young theorem; at line 359, change the heading to ## Full and reduced SVDs; at line 509, change the heading to ## Polar decomposition; at line 537, change the section heading to ## Application: PCA; and at line 672, change the heading to ## PCA with eigenvalues and eigenvectors.

• [qe-writing-008] Remove the excessive spaces at line 19,27,29; line 31-59; line 63-125; line 129-274; line 326-349; line 359-506; line 509-535; line 539-620; line 662,665,668; line 674,680,724,735-736,747,753,755,759,761; line 770,788,841,944; and line 1037,1039,1041.

• [qe-math-001] At line 696, write $m\times m$ rather than using the Unicode multiplication sign.

• [qe-math-002] At line 750,757, use $\top$ rather than $T$ for transposes.

• [qe-code-001] At line 300, add the missing space after the comma in np.linalg.svd(A, full_matrices=True); at line 427-506, apply standard spacing around commas, assignments, and matrix multiplication in the code cells; and at line 891, remove the extra space after the comma in the SVD assignment.

• [qe-fig-001] At line 962,973, remove the two fixed figsize=(14, 5) settings.

• [qe-fig-003] At line 963-979, remove the two suptitle calls and four set_title calls.

• [qe-fig-008] At line 964-978, add lw=2 to all four line plots.

• [qe-link-001] At line 1037-1041, use doc links for lecture references.

• [qe-admon-004] At line 472-479, replace **Remarks:** and its associated paragraphs with a {prf:remark} directive.

What do you think? Happy to put up a PR.

Best,
Longye

Start with the svd_intro lecture and inspect the referenced line ranges, especially the NumPy SVD examples and the DecomAnalysis docstring. Apply the listed terminology, notation, dimensionality, PCA, and centering corrections, then verify that the lecture consistently matches the stated NumPy outputs and formulas.