9.4 Inner Product Spaces, Gram-Schmidt, and the Spectral Theorem

9.4 Inner Product Spaces, Gram-Schmidt, and the Spectral Theorem

Inner product spaces enrich vector spaces with metric concepts including lengths, angles, and orthogonality. On the GRE Mathematics Subject Test, questions test the Cauchy-Schwarz inequality, Gram-Schmidt orthogonalization, QR factorization, the Spectral Theorem for symmetric and Hermitian matrices, and Sylvester's criterion for quadratic form definiteness.

---

Inner Products, Norms, and Geometric Inequalities

An inner product on a real vector space $V$ is a map $\langle \cdot, \cdot \rangle: V \times V \to \mathbb{R}$ satisfying:

1. Symmetry: $\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$.
2. Bilinearity: $\langle c\mathbf{u}_1 + d\mathbf{u}_2, \mathbf{v} \rangle = c\langle \mathbf{u}_1, \mathbf{v} \rangle + d\langle \mathbf{u}_2, \mathbf{v} \rangle$.
3. Positive Definiteness: $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$, with $\langle \mathbf{v}, \mathbf{v} \rangle = 0 \iff \mathbf{v} = \mathbf{0}$.

For a complex vector space, symmetry is replaced by conjugate symmetry $\langle \mathbf{u}, \mathbf{v} \rangle = \overline{\langle \mathbf{v}, \mathbf{u} \rangle}$, making the inner product sesquilinear.

The Induced Norm and Inequalities

The inner product induces the Euclidean norm $|\mathbf{v}| = \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle}$. Foundational inequalities include:

• Cauchy-Schwarz Inequality: $|\langle \mathbf{u}, \mathbf{v} \rangle| \le \|\mathbf{u}\| \|\mathbf{v}\|,$ with equality if and only if $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent.
• Triangle Inequality: $|\mathbf{u} + \mathbf{v}| \le |\mathbf{u}| + |\mathbf{v}|$.
• Parallelogram Identity: $|\mathbf{u} + \mathbf{v}|^2 + |\mathbf{u} - \mathbf{v}|^2 = 2|\mathbf{u}|^2 + 2|\mathbf{v}|^2$.
• Pythagorean Theorem: If $\langle \mathbf{u}, \mathbf{v} \rangle = 0$, then $|\mathbf{u} + \mathbf{v}|^2 = |\mathbf{u}|^2 + |\mathbf{v}|^2$.

---

The Gram-Schmidt Process and QR Factorization

Given a linearly independent set ${\mathbf{v}_1, \dots, \mathbf{v}_k}$, the Gram-Schmidt process generates an orthogonal set ${\mathbf{u}_1, \dots, \mathbf{u}_k}$ spanning the same subspace: $\begin{aligned} \mathbf{u}_1 &= \mathbf{v}_1 \\ \mathbf{u}_2 &= \mathbf{v}_2 - \frac{\langle \mathbf{v}_2, \mathbf{u}_1 \rangle}{\|\mathbf{u}_1\|^2} \mathbf{u}_1 \\ &\;\;\vdots \\ \mathbf{u}_k &= \mathbf{v}_k - \sum_{j=1}^{k-1} \frac{\langle \mathbf{v}_k, \mathbf{u}_j \rangle}{\|\mathbf{u}_j\|^2} \mathbf{u}_j \end{aligned}$ Normalizing each vector $\mathbf{e}_i = \mathbf{u}_i / |\mathbf{u}_i|$ produces an orthonormal basis ($\langle \mathbf{e}_i, \mathbf{e}j \rangle = \delta{ij}$).

QR Factorization

Every $m \times n$ matrix $A$ with linearly independent columns admits a $QR$ factorization: $A = Q R,$ where $Q \in M_{m \times n}(\mathbb{R})$ has orthonormal columns ($Q^T Q = I_n$), and $R \in M_{n \times n}(\mathbb{R})$ is an upper triangular matrix with strictly positive diagonal entries $r_{ii} = |\mathbf{u}_i| > 0$.

---

Orthogonal Projections and Subspaces

Let $W \subset V$ be a finite-dimensional subspace with orthonormal basis ${\mathbf{e}_1, \dots, \mathbf{e}_k}$.

• The orthogonal projection of $\mathbf{v} \in V$ onto $W$ is: $P_W(\mathbf{v}) = \sum_{i=1}^k \langle \mathbf{v}, \mathbf{e}_i \rangle \mathbf{e}_i.$
• The vector $\mathbf{v} - P_W(\mathbf{v})$ lies in the orthogonal complement $W^\perp = {\mathbf{x} \in V : \langle \mathbf{x}, \mathbf{w} \rangle = 0, ; \forall \mathbf{w} \in W}$.
• Best Approximation Theorem: $P_W(\mathbf{v})$ is the unique vector in $W$ minimizing the distance to $\mathbf{v}$: $\|\mathbf{v} - P_W(\mathbf{v})\| = \min_{\mathbf{w} \in W} \|\mathbf{v} - \mathbf{w}\|.$

---

Symmetric and Hermitian Matrices & The Spectral Theorem

A real matrix $A$ is symmetric if $A = A^T$. A complex matrix $A$ is Hermitian if $A = A^* = \overline{A}^T$.

Fundamental Properties of Symmetric/Hermitian Operators

1. Real Spectra: Every eigenvalue of a real symmetric or complex Hermitian matrix is strictly real. Proof: If $A\mathbf{v} = \lambda \mathbf{v}$ with $\mathbf{v} \neq \mathbf{0}$, then $\lambda |\mathbf{v}|^2 = \mathbf{v}^* A \mathbf{v} = (A\mathbf{v})^* \mathbf{v} = \overline{\lambda} |\mathbf{v}|^2 \implies \lambda = \overline{\lambda} \in \mathbb{R}$.
2. Orthogonal Eigenspaces: Eigenvectors corresponding to distinct eigenvalues are mutually orthogonal. Proof: If $A\mathbf{v}_1 = \lambda_1 \mathbf{v}_1$ and $A\mathbf{v}_2 = \lambda_2 \mathbf{v}_2$ with $\lambda_1 \neq \lambda_2$, then $\lambda_1 \langle \mathbf{v}_1, \mathbf{v}_2 \rangle = \langle A\mathbf{v}_1, \mathbf{v}_2 \rangle = \langle \mathbf{v}_1, A\mathbf{v}_2 \rangle = \lambda_2 \langle \mathbf{v}_1, \mathbf{v}_2 \rangle$. Because $\lambda_1 \neq \lambda_2$, we must have $\langle \mathbf{v}_1, \mathbf{v}_2 \rangle = 0$.

The Spectral Theorem

Let $A \in M_{n \times n}(\mathbb{R})$ be a real symmetric matrix. Then:

1. $A$ has $n$ real eigenvalues (counted with algebraic multiplicity).
2. There exists an orthonormal basis of $\mathbb{R}^n$ consisting of eigenvectors of $A$.
3. $A$ is orthogonally diagonalizable: $A = Q \Lambda Q^T,$ where $Q$ is an orthogonal matrix ($Q^{-1} = Q^T$) and $\Lambda = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$.

In terms of rank-1 projection matrices, $A = \sum_{i=1}^n \lambda_i \mathbf{q}_i \mathbf{q}_i^T$.

---

Quadratic Forms and Definiteness

A quadratic form on $\mathbb{R}^n$ is a polynomial $q(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}$, where $A$ is a real symmetric matrix.

Classification by Eigenvalues

Because $A = Q \Lambda Q^T$, setting $\mathbf{y} = Q^T \mathbf{x}$ transforms the form into a sum of squares: $q(\mathbf{x}) = \mathbf{y}^T \Lambda \mathbf{y} = \sum_{i=1}^n \lambda_i y_i^2.$

---

Step-by-Step Worked Problems

Problem 1: Gram-Schmidt and QR Factorization

Apply Gram-Schmidt to orthogonalize $\mathbf{v}_1 = (1, 1, 0)^T$ and $\mathbf{v}_2 = (1, 0, 2)^T$.

Solution:

1. First vector: $\mathbf{u}_1 = \mathbf{v}_1 = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}, \quad \|\mathbf{u}_1\|^2 = 1^2 + 1^2 + 0^2 = 2.$
2. Second vector projection: $\langle \mathbf{v}_2, \mathbf{u}_1 \rangle = 1(1) + 0(1) + 2(0) = 1.$ $\mathbf{u}_2 = \mathbf{v}_2 - \frac{\langle \mathbf{v}_2, \mathbf{u}_1 \rangle}{\|\mathbf{u}_1\|^2} \mathbf{u}_1 = \begin{pmatrix} 1 \\ 0 \\ 2 \end{pmatrix} - \frac{1}{2} \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 1/2 \\ -1/2 \\ 2 \end{pmatrix}.$
3. Norm of $\mathbf{u}_2$: $\|\mathbf{u}_2\|^2 = (1/2)^2 + (-1/2)^2 + 2^2 = 1/4 + 1/4 + 4 = 9/2 \implies \|\mathbf{u}_2\| = \frac{3}{\sqrt{2}}.$
4. Orthonormal basis vectors: $\mathbf{e}_1 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}, \quad \mathbf{e}_2 = \frac{\sqrt{2}}{3} \begin{pmatrix} 1/2 \\ -1/2 \\ 2 \end{pmatrix} = \frac{1}{3\sqrt{2}} \begin{pmatrix} 1 \\ -1 \\ 4 \end{pmatrix}.$

Problem 2: Definiteness via Sylvester's Criterion

Classify the quadratic form $q(x_1, x_2, x_3) = 2x_1^2 + 2x_2^2 + 2x_3^2 - 2x_1 x_2 - 2x_2 x_3$.

Solution:

1. Construct the associated symmetric matrix $A$: $A = \begin{pmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{pmatrix}$
2. Compute leading principal minors:
   • $\Delta_1 = \det(2) = 2 > 0$.
   • $\Delta_2 = \det \begin{pmatrix} 2 & -1 \ -1 & 2 \end{pmatrix} = 4 - 1 = 3 > 0$.
   • $\Delta_3 = \det(A) = 2(4 - 1) - (-1)(-2 - 0) + 0 = 2(3) - 2 = 6 - 2 = 4 > 0$.
3. Because all leading principal minors are strictly positive ($\Delta_1 = 2 > 0, \Delta_2 = 3 > 0, \Delta_3 = 4 > 0$), Sylvester's Criterion confirms that $A$ is positive definite.

---

GRE Exam Traps & Pitfalls

Trap 1: Misapplying Sylvester's Criterion to Semidefinite Forms Leading principal minors $\Delta_k \ge 0$ do NOT guarantee positive semidefiniteness! Consider $A = \begin{pmatrix} 0 & 0 \ 0 & -1 \end{pmatrix}$. Here $\Delta_1 = 0$ and $\Delta_2 = 0 \ge 0$, yet $A$ has eigenvalue $-1$ and is indefinite. To test semidefiniteness, you must check all principal minors or compute the eigenvalues directly.

Trap 2: Normalization Omission in Gram-Schmidt Projections Forgetting to divide by $|\mathbf{u}_j|^2$ when calculating $\frac{\langle \mathbf{v}, \mathbf{u}_j \rangle}{|\mathbf{u}_j|^2} \mathbf{u}_j$ distorts the orthogonal basis and destroys orthogonality.

Trap 3: Overlooking Complex Conjugation in Hermitian Matrices For complex vectors, $\langle \mathbf{u}, \mathbf{v} \rangle = \sum u_i \overline{v_i}$. Forgetting conjugation in $\mathbf{v}^* A \mathbf{v}$ leads to incorrect, non-real expressions.