Unit 7: Inner Product Spaces, Gram-Schmidt, Adjoint Operators & Spectral Theory

1. Definition of Inner Product Spaces

Let $V$ be a vector space over the field $F$ (where $F = \mathbb{R}$ or $F = \mathbb{C}$). An Inner Product on $V$ is a function $\langle \cdot, \cdot \rangle: V \times V \to F$ satisfying four fundamental axioms for all $\vec{u}, \vec{v}, \vec{w} \in V$ and all scalars $c \in F$:

1. Conjugate Symmetry (Hermitian Symmetry):

(If $F = \mathbb{R}$, this reduces to ordinary symmetry: $\langle \vec{u}, \vec{v} \rangle = \langle \vec{v}, \vec{u} \rangle$.)

2. Linearity in the First Argument:

• Additivity: $\langle \vec{u} + \vec{v}, \vec{w} \rangle = \langle \vec{u}, \vec{w} \rangle + \langle \vec{v}, \vec{w} \rangle$
• Homogeneity: $\langle c\vec{u}, \vec{v} \rangle = c \langle \vec{u}, \vec{v} \rangle$

3. Conjugate Linearity (Anti-Linearity) in the Second Argument:

Proof: $\langle \vec{u}, c\vec{v} \rangle = \overline{\langle c\vec{v}, \vec{u} \rangle} = \overline{c \langle \vec{v}, \vec{u} \rangle} = \overline{c} \, \overline{\langle \vec{v}, \vec{u} \rangle} = \overline{c} \langle \vec{u}, \vec{v} \rangle$.

4. Positive-Definiteness:

A vector space $V$ equipped with an inner product is called an Inner Product Space (or a Pre-Hilbert Space; if complete, a Hilbert Space). A real inner product space is often called a Euclidean Space, and a complex inner product space a Unitary Space.

---

2. Archetypal Inner Product Spaces

(a) Euclidean Space $\mathbb{R}^n$:

(b) Complex Unitary Space $\mathbb{C}^n$:

(Note: In mathematical physics literature, the convention $\langle \vec{x}, \vec{y} \rangle = \sum \overline{x_i} y_i$ is also standard; both conventions satisfy the axioms up to transposition).

(c) Continuous Function Space $C[a, b]$:

Positivity holds because if $f$ is continuous and $\int_a^b |f(t)|^2 dt = 0$, then $f(t) = 0$ identically on $[a, b]$.

(d) Matrix Space $M_{m \times n}(\mathbb{C})$ (Frobenius Inner Product):

---

3. Induced Norm and Metric

For any vector $\vec{v} \in V$, the Induced Norm (or length) is defined by:

The induced metric (distance function) between vectors $\vec{u}, \vec{v}$ is:

A vector $\vec{u}$ is called a unit vector if $\|\vec{u}\| = 1$. Any non-zero vector $\vec{v}$ can be normalized:

---

4. The Cauchy-Schwarz Inequality

Theorem 7.1 (Cauchy-Schwarz Inequality):

For any vectors $\vec{u}, \vec{v}$ in an inner product space $V$:

Equality holds if and only if $\vec{u}$ and $\vec{v}$ are linearly dependent (i.e., one is a scalar multiple of the other).

Rigorous Proof:

• Case 1: If $\vec{v} = \vec{0}$, then $\langle \vec{u}, \vec{0} \rangle = 0$ and $\|\vec{v}\| = 0$, so both sides equal $0$. The inequality holds as an equality, and $\{\vec{u}, \vec{0}\}$ is linearly dependent.
• Case 2: Assume $\vec{v} \neq \vec{0}$. Let $c \in F$ be an arbitrary scalar. By positive-definiteness:

Expanding the inner product using linearity and conjugate symmetry:

Now make the specific choice of scalar:

Substituting this $c$:

Multiplying through by $\|\vec{v}\|^2 > 0$:

Taking the non-negative square root of both sides gives:

Equality holds if and only if $\|\vec{u} - c\vec{v}\|^2 = 0 \iff \vec{u} - c\vec{v} = \vec{0} \iff \vec{u} = c\vec{v}$, which means $\vec{u}$ and $\vec{v}$ are collinear. $\blacksquare$

---

5. Consequences of Cauchy-Schwarz

(a) The Triangle Inequality (Minkowski's Inequality):

For all $\vec{u}, \vec{v} \in V$:

Proof:

Taking square roots yields $\|\vec{u} + \vec{v}\| \le \|\vec{u}\| + \|\vec{v}\|$. $\blacksquare$

(b) The Parallelogram Law:

This geometric identity characterizes inner product norms: by the Jordan-von Neumann theorem, a normed space is an inner product space if and only if its norm satisfies the parallelogram identity.

(c) The Pythagorean Theorem:

Two vectors $\vec{u}, \vec{v}$ are orthogonal ($\vec{u} \perp \vec{v}$) if $\langle \vec{u}, \vec{v} \rangle = 0$. If $\vec{u} \perp \vec{v}$, then:

### 1. Orthogonal and Orthonormal Systems A subset $S = \{\vec{v}_1, \vec{v}_2, \dots, \vec{v}_k\} \subset V$ is called: - **Orthogonal** if $\langle \vec{v}_i, \vec{v}_j \rangle = 0$ for all $i \neq j$. - **Orthonormal** if it is orthogonal and every vector is normalized: $$\langle \vec{e}_i, \vec{e}_j \rangle = \delta_{ij} = \begin{cases} 1, & i = j \\ 0, & i \neq j \end{cases}$$ #### Theorem 7.2 (Linear Independence of Orthogonal Sets): Any orthogonal set of non-zero vectors $S = \{\vec{v}_1, \vec{v}_2, \dots, \vec{v}_k\}$ in an inner product space $V$ is linearly independent. *Proof:* Suppose $c_1 \vec{v}_1 + c_2 \vec{v}_2 + \cdots + c_k \vec{v}_k = \vec{0}$. Take the inner product of both sides with $\vec{v}_j$ ($1 \le j \le k$): $$\left\langle \sum_{i=1}^k c_i \vec{v}_i, \vec{v}_j \right\rangle = \langle \vec{0}, \vec{v}_j \rangle = 0 \implies \sum_{i=1}^k c_i \langle \vec{v}_i, \vec{v}_j \rangle = 0$$ Since $\langle \vec{v}_i, \vec{v}_j \rangle = 0$ for $i \neq j$, the sum collapses to a single non-zero term: $$c_j \langle \vec{v}_j, \vec{v}_j \rangle = c_j \|\vec{v}_j\|^2 = 0$$ Since $\vec{v}_j \neq \vec{0}$, $\|\vec{v}_j\|^2 > 0$, forcing $c_j = 0$. Since this holds for all $j \in \{1, \dots, k\}$, the set is linearly independent. $\blacksquare$ --- ### 2. Orthogonal Complements and the Orthogonal Projection Theorem #### Orthogonal Complement: Let $W$ be a subspace of an inner product space $V$. The **orthogonal complement** of $W$, denoted $W^\perp$ ("$W$ perp"), is defined by: $$W^\perp = \{\vec{v} \in V : \langle \vec{v}, \vec{w} \rangle = 0 \text{ for all } \vec{w} \in W\}$$ #### Fundamental Properties of $W^\perp$: 1. $W^\perp$ is a subspace of $V$. 2. $W \cap W^\perp = \{\vec{0}\}$ (if $\vec{v} \in W \cap W^\perp$, then $\langle \vec{v}, \vec{v} \rangle = 0 \implies \vec{v} = \vec{0}$). 3. In finite dimensions: $\dim W + \dim W^\perp = \dim V$. 4. $(W^\perp)^\perp = W$. #### Theorem 7.3 (Orthogonal Decomposition Theorem): Let $W$ be a finite-dimensional subspace of an inner product space $V$. Then every vector $\vec{v} \in V$ can be uniquely written as: $$\vec{v} = \vec{w} + \vec{w}^\perp \quad \text{where } \vec{w} \in W \text{ and } \vec{w}^\perp \in W^\perp$$ That is, $V = W \oplus W^\perp$. The vector $\vec{w}$ is called the **orthogonal projection** of $\vec{v}$ onto $W$, written $\text{proj}_W(\vec{v})$. If $\{\vec{e}_1, \dots, \vec{e}_k\}$ is an **orthonormal basis** for $W$, then: $$\text{proj}_W(\vec{v}) = \sum_{i=1}^k \langle \vec{v}, \vec{e}_i \rangle \vec{e}_i$$ The projection satisfies the **Best Approximation Property**: $$\|\vec{v} - \text{proj}_W(\vec{v})\| \le \|\vec{v} - \vec{w}\| \quad \text{for all } \vec{w} \in W$$ with strict inequality for any $\vec{w} \neq \text{proj}_W(\vec{v})$. --- --- ### 3. The Gram-Schmidt Orthogonalization Algorithm Let $\{\vec{v}_1, \vec{v}_2, \dots, \vec{v}_n\}$ be a linearly independent set in an inner product space $V$. The **Gram-Schmidt Process** constructs an orthogonal set $\{\vec{u}_1, \vec{u}_2, \dots, \vec{u}_n\}$ such that: $$\text{span}\{\vec{u}_1, \dots, \vec{u}_k\} = \text{span}\{\vec{v}_1, \dots, \vec{v}_k\} \quad \text{for all } k = 1, 2, \dots, n$$ #### Step-by-Step Construction: $$\begin{aligned} \vec{u}_1 &= \vec{v}_1 \ \vec{u}_2 &= \vec{v}_2 - \frac{\langle \vec{v}_2, \vec{u}_1 \rangle}{\|\vec{u}_1\|^2} \vec{u}_1 \ \vec{u}_3 &= \vec{v}_3 - \frac{\langle \vec{v}_3, \vec{u}_1 \rangle}{\|\vec{u}_1\|^2} \vec{u}_1 - \frac{\langle \vec{v}_3, \vec{u}_2 \rangle}{\|\vec{u}_2\|^2} \vec{u}_2 \ &\quad \vdots \ \vec{u}_k &= \vec{v}_k - \sum_{j=1}^{k-1} \frac{\langle \vec{v}_k, \vec{u}_j \rangle}{\|\vec{u}_j\|^2} \vec{u}_j \end{aligned}$$ #### Orthonormalization: To obtain an **orthonormal basis** $\{\vec{e}_1, \dots, \vec{e}_n\}$, normalize each vector: $$\vec{e}_k = \frac{\vec{u}_k}{\|\vec{u}_k\|}, \quad k = 1, 2, \dots, n$$ --- ### 4. The QR Factorization Let $A \in \mathbb{R}^{m \times n}$ be an $m \times n$ matrix with linearly independent columns $\vec{a}_1, \dots, \vec{a}_n$ ($m \ge n$). Applying Gram-Schmidt to the columns of $A$ yields orthonormal vectors $\vec{q}_1, \dots, \vec{q}_n$. Each original column vector can be written as: $$\vec{a}_k = \sum_{j=1}^k \langle \vec{a}_k, \vec{q}_j \rangle \vec{q}_j = r_{1k}\vec{q}_1 + r_{2k}\vec{q}_2 + \cdots + r_{kk}\vec{q}_k$$ In matrix notation: $$A = Q R$$ where: - $Q \in \mathbb{R}^{m \times n}$ has orthonormal columns ($Q^T Q = I_n$). - $R \in \mathbb{R}^{n \times n}$ is upper triangular with positive diagonal entries $r_{kk} = \|\vec{u}_k\| > 0$: $$R = \begin{pmatrix} \langle \vec{a}_1, \vec{q}_1 \rangle & \langle \vec{a}_2, \vec{q}_1 \rangle & \cdots & \langle \vec{a}_n, \vec{q}_1 \rangle \\ 0 & \langle \vec{a}_2, \vec{q}_2 \rangle & \cdots & \langle \vec{a}_n, \vec{q}_2 \rangle \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \langle \vec{a}_n, \vec{q}_n \rangle \end{pmatrix}$$ #### Application to Least Squares: The overdetermined system $A\vec{x} = \vec{b}$ has normal equations: $$A^T A \hat{x} = A^T \vec{b} \implies (R^T Q^T)(Q R)\hat{x} = R^T Q^T \vec{b} \implies R^T R \hat{x} = R^T Q^T \vec{b}$$ Since $R$ is invertible ($r_{kk} > 0$), this simplifies directly to the upper-triangular system: $$R \hat{x} = Q^T \vec{b}$$ which can be solved by simple back-substitution without computing $A^T A$ or its inverse! --- ### 5. Bessel's Inequality and Parseval's Identity Let $\{\vec{e}_1, \vec{e}_2, \dots\}$ be an orthonormal sequence in an inner product space $V$. For any $\vec{v} \in V$: - **Bessel's Inequality:** $$\sum_{i=1}^k |\langle \vec{v}, \vec{e}_i \rangle|^2 \le \|\vec{v}\|^2$$ - **Parseval's Identity (Completeness):** If $\{\vec{e}_i\}$ is a complete orthonormal basis: $$\|\vec{v}\|^2 = \sum_{i=1}^n |\langle \vec{v}, \vec{e}_i \rangle|^2 \quad \text{and} \quad \langle \vec{u}, \vec{v} \rangle = \sum_{i=1}^n \langle \vec{u}, \vec{e}_i \rangle \overline{\langle \vec{v}, \vec{e}_i \rangle}$$

1. Dual Spaces and the Riesz Representation Theorem

Let $V$ be a vector space over field $F$. The Dual Space $V^*$ is the space of all linear functionals on $V$:

If $\beta = \{\vec{v}_1, \dots, \vec{v}_n\}$ is a basis of $V$, its dual basis $\beta^ = \{f_1, \dots, f_n\} \subset V^$ is uniquely determined by:

Every functional $f \in V^*$ can be uniquely expanded as $f = \sum_{i=1}^n f(\vec{v}_i) f_i$.

Theorem 7.4 (Riesz Representation Theorem for Finite Dimensions):

Let $V$ be a finite-dimensional inner product space over $F$. For every linear functional $f \in V^*$, there exists a unique vector $\vec{z} \in V$ such that:

Moreover, $\|f\|_{V^*} = \|\vec{z}\|$.

Proof:

• Existence: Choose an orthonormal basis $\{\vec{e}_1, \dots, \vec{e}_n\}$ of $V$.

Define the candidate vector:

Then for any basis vector $\vec{e}_j$:

Since $f$ and the functional $\vec{v} \mapsto \langle \vec{v}, \vec{z} \rangle$ agree on a basis, by linearity they agree on all of $V$: $f(\vec{v}) = \langle \vec{v}, \vec{z} \rangle$ for all $\vec{v} \in V$.

• Uniqueness: Suppose there exists another vector $\vec{z}' \in V$ such that $f(\vec{v}) = \langle \vec{v}, \vec{z}' \rangle$ for all $\vec{v}$.

Then $\langle \vec{v}, \vec{z} - \vec{z}' \rangle = 0$ for all $\vec{v} \in V$. Choosing $\vec{v} = \vec{z} - \vec{z}'$ gives $\|\vec{z} - \vec{z}'\|^2 = 0 \implies \vec{z} = \vec{z}'$. $\blacksquare$

---

2. The Adjoint of a Linear Operator

Let $T: V \to W$ be a linear transformation between finite-dimensional inner product spaces over $F$. For each fixed $\vec{w} \in W$, the map $\vec{v} \mapsto \langle T(\vec{v}), \vec{w} \rangle_W$ is a linear functional on $V$. By the Riesz Representation Theorem, there exists a unique vector in $V$, denoted $T^*(\vec{w})$, such that:

The mapping $T^*: W \to V$ is linear and is called the Adjoint (or Hermitian Adjoint) of $T$.

Fundamental Algebraic Properties:

1. $(S + T)^ = S^ + T^*$
2. $(c T)^ = \bar{c} T^$
3. $(S T)^ = T^ S^*$
4. $(T^)^ = T$
5. If $T$ is invertible, $(T^{-1})^ = (T^)^{-1}$.
6. $\|T^*\| = \|T\|$.

Matrix Representation:

If $\beta$ is an orthonormal basis of $V$ and $\gamma$ is an orthonormal basis of $W$, then:

The matrix of the adjoint is the conjugate transpose (Hermitian conjugate) of the matrix of $T$.

Fundamental Subspace Relations for Adjoints:

• $\ker(T^*) = (\text{im}(T))^\perp$
• $\text{im}(T^*) = (\ker(T))^\perp$
• $\ker(T) = (\text{im}(T^*))^\perp$
• $\text{im}(T) = (\ker(T^*))^\perp$

---

3. Special Classes of Operators

| Operator Class | Defining Relation | Matrix Property (ONB) | Eigenvalue Character | | :--- | :--- | :--- | :--- | | Self-Adjoint (Hermitian) | $T^* = T$ | $A^\dagger = A$ | All $\lambda_i \in \mathbb{R}$ (strictly real) | | Skew-Hermitian | $T^* = -T$ | $A^\dagger = -A$ | All $\lambda_i \in i\mathbb{R}$ (purely imaginary) | | Unitary / Orthogonal | $T^ T = T T^ = I$ | $U^\dagger U = I$ | All $|\lambda_i| = 1$ (on the unit circle) | | Normal | $T T^ = T^ T$ | $A A^\dagger = A^\dagger A$ | Diagonalizable with ONB of eigenvectors | | Projection (Orthogonal) | $P^2 = P = P^*$ | $P^\dagger = P = P^2$ | $\lambda \in \{0, 1\}$ |

Geometric Properties of Unitary / Orthogonal Operators:

An operator $U: V \to V$ is unitary (orthogonal if $F = \mathbb{R}$) if and only if it preserves the inner product:

Consequently, unitary operators preserve norms ($\|U\vec{v}\| = \|\vec{v}\|$) and distances ($d(U\vec{u}, U\vec{v}) = d(\vec{u}, \vec{v})$)—they are isometries of the inner product space.

1. Fundamental Lemmas for Normal and Self-Adjoint Operators

Lemma 7.1 (Eigenvalues of Self-Adjoint Operators are Real):

Let $T: V \to V$ be a self-adjoint operator on an inner product space ($T^* = T$). Then every eigenvalue $\lambda$ of $T$ is real.

Proof: Let $\vec{v} \neq \vec{0}$ be an eigenvector of $T$ with eigenvalue $\lambda$, so $T(\vec{v}) = \lambda \vec{v}$.

Using the self-adjoint property:

Equating the first and last expressions:

Since $\vec{v} \neq \vec{0}$, $\|\vec{v}\|^2 > 0$. Therefore $\lambda - \bar{\lambda} = 0 \implies \lambda = \bar{\lambda}$, so $\lambda \in \mathbb{R}$. $\blacksquare$

Lemma 7.2 (Eigenvectors of Normal Operators are Orthogonal):

If $T$ is a normal operator ($T T^ = T^ T$), and $\vec{u}, \vec{v}$ are eigenvectors corresponding to distinct eigenvalues $\lambda_1 \neq \lambda_2$, then $\vec{u} \perp \vec{v}$.

Proof: First, observe that for a normal operator, $\|(T - \lambda I)\vec{x}\| = \|(T^* - \bar{\lambda} I)\vec{x}\|$ for all $\vec{x}$. Indeed:

Thus $T\vec{v} = \lambda_2 \vec{v} \iff T^*\vec{v} = \bar{\lambda}_2 \vec{v}$. Now consider $\langle T\vec{u}, \vec{v} \rangle$:

Rearranging:

Since $\lambda_1 \neq \lambda_2$, it follows immediately that $\langle \vec{u}, \vec{v} \rangle = 0$. $\blacksquare$

---

2. The Spectral Theorem

Theorem 7.5 (The Complex Spectral Theorem):

Let $V$ be a finite-dimensional complex inner product space, and let $T: V \to V$ be a linear operator. Then $T$ is normal ($T T^ = T^ T$) if and only if there exists an orthonormal basis of $V$ consisting of eigenvectors of $T$.

In matrix terms: A complex matrix $A \in M_n(\mathbb{C})$ is normal ($A A^\dagger = A^\dagger A$) if and only if it is unitarily diagonalizable:

where $U$ is unitary ($U^\dagger U = I_n$) and $\Lambda$ is diagonal.

Theorem 7.6 (The Real Spectral Theorem):

Let $V$ be a finite-dimensional real inner product space, and let $T: V \to V$ be a linear operator. Then $T$ is self-adjoint ($T^* = T$) if and only if there exists an orthonormal basis of $V$ consisting of eigenvectors of $T$.

In matrix terms: A real matrix $A \in M_n(\mathbb{R})$ is symmetric ($A^T = A$) if and only if it is orthogonally diagonalizable:

where $Q$ is an orthogonal matrix ($Q^T Q = I_n$) and $\Lambda$ is a real diagonal matrix:

---

3. The Spectral Decomposition (Resolution of the Identity)

Let $T$ be a normal operator on $V$ with distinct eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_k$, and let $E_i = \ker(T - \lambda_i I)$ be the corresponding eigenspaces. Because $T$ is normal:

Let $P_i$ denote the orthogonal projection of $V$ onto the eigenspace $E_i$.

Properties of the Spectral Projectors $\{P_1, \dots, P_k\}$:

1. Idempotence & Self-Adjointness:

2. Mutual Orthogonality:

3. Resolution of the Identity (Completeness):

4. Spectral Decomposition of $T$:

Functional Calculus:

For any polynomial, rational function, or analytical function $f(z)$ defined on the spectrum of $T$:

In particular:

• Powers: $T^m = \sum_{i=1}^k \lambda_i^m P_i$
• Inverse: $T^{-1} = \sum_{i=1}^k \lambda_i^{-1} P_i$ (if $\lambda_i \neq 0$ for all $i$)
• Matrix Exponential: $e^{t T} = \sum_{i=1}^k e^{t \lambda_i} P_i$