Unit 5: Initial Value Problems for Ordinary Differential Equations: Single-Step Methods

1. The General Initial Value Problem (IVP)

Consider the first-order ordinary differential equation with prescribed initial state:

where $y: [t_0, T] \to \mathbb{R}^d$ and $f: [t_0, T] \times \mathbb{R}^d \to \mathbb{R}^d$.

Equivalent Volterra Integral Equation:

Integrating both sides over $[t_0, t]$ yields:

This integral formulation converts the differential equation into a fixed-point problem over the Banach space $C([t_0, T], \mathbb{R}^d)$.

2. The Picard-Lindelöf Existence and Uniqueness Theorem

Theorem (Picard-Lindelöf): Let $D = [t_0 - a, t_0 + a] \times \overline{B}(y_0, b) \subset \mathbb{R} \times \mathbb{R}^d$. Suppose:

1. $f(t, y)$ is continuous on $D$, with $M = \max_{(t, y) \in D} \|f(t, y)\| < \infty$.
2. $f(t, y)$ satisfies a uniform Lipschitz condition with respect to $y$ on $D$:

Then there exists a unique continuously differentiable solution $y(t)$ to the IVP on the interval $I = [t_0 - h, t_0 + h]$, where $h = \min\left(a, \frac{b}{M}\right)$.

Picard's Method of Successive Approximations:

Define the operator $T: C(I) \to C(I)$ by:

Starting from the constant initial guess $y^{(0)}(t) \equiv y_0$, generate the sequence:

Proof of Uniform Convergence:

For any $t \in [t_0, t_0 + h]$:

By induction:

Since the infinite series:

converges, the Weierstrass M-test guarantees that the sequence of continuous functions $\{y^{(k)}(t)\}$ converges uniformly on $I$ to a unique limit $y^(t) = \lim_{k \to \infty} y^{(k)}(t)$. Taking the limit under the integral sign proves $y^(t) = T y^*(t)$.

1. High-Order Taylor Series Method

Assuming $f(t, y)$ is $p$-times continuously differentiable, the exact solution $y(t)$ can be expanded about $t_n$ using Taylor's theorem:

where $\xi_n \in (t_n, t_{n+1})$.

Using the chain rule, total derivatives of $y(t)$ are evaluated directly from the governing ODE:

The Taylor Method of Order $p$:

where the increment function is:

Fatal Practical Limitation: For systems of ODEs or complex nonlinear functions $f$, computing high-order total derivatives analytically requires nested symbolic chain rules whose algebraic complexity explodes exponentially (the "curse of differentiation").

2. Forward Euler's Method ($p = 1$)

Truncating after the linear term yields the simplest numerical integrator:

Error Analysis:

• Local Truncation Error (LTE):

• Local Error per unit step: $\tau_{n+1} = \frac{d_{n+1}}{h} = \mathcal{O}(h)$.
• Global Truncation Error (GTE): Over a fixed interval $[t_0, T]$ with $N = (T - t_0)/h$ steps, errors accumulate:

Hence, Forward Euler is only first-order accurate. To halve the error, computational work must double.

1. General Philosophy of Runge-Kutta Methods

The core innovation of Carl Runge (1895) and Wilhelm Kutta (1901) was to match the Taylor series expansion up to order $p$ using only evaluations of $f(t, y)$ at intermediate points, completely bypassing analytic differentiation.

An explicit $s$-stage Runge-Kutta (ERK) method advances from $(t_n, y_n)$ to $t_{n+1} = t_n + h$ via:

where stage derivatives $k_i$ are sampled sequentially:

2. The Butcher Tableau Representation

John C. Butcher formalized Runge-Kutta schemes into a compact matrix-vector tableau:

where $c_i = \sum_{j=1}^{i-1} a_{ij}$ ensures internal stage consistency (zero-order consistency for autonomous systems).

3. Family of Two-Stage Second-Order Runge-Kutta Methods (RK2)

For $s = 2$:

Expanding in Bivariate Taylor Series:

Expanding $k_2 = f(t_n + c_2 h, y_n + h a_{21} f)$ about $(t_n, y_n)$:

Substituting into the numerical update:

Comparing with the exact Taylor series:

Equating coefficients of powers of $h$ and elementary differentials:

1. $h^1$ term: $b_1 + b_2 = 1$
2. $h^2 f_t$ term: $b_2 c_2 = \frac{1}{2}$
3. $h^2 f f_y$ term: $b_2 a_{21} = \frac{1}{2}$

This gives 3 nonlinear equations in 4 unknowns ($b_1, b_2, c_2, a_{21}$), yielding a one-parameter family of second-order methods where $a_{21} = c_2$ and $b_2 = \frac{1}{2 c_2}$, $b_1 = 1 - \frac{1}{2 c_2}$ for any $c_2 \ne 0$:

Famous Members of the RK2 Family:

1. Explicit Midpoint Method ($c_2 = 1/2$):

2. Heun's Method / Improved Euler ($c_2 = 1$):

3. Ralston's Method ($c_2 = 2/3$):

Minimizes the truncation error bound: $b_1 = 1/4$, $b_2 = 3/4$, $a_{21} = 2/3$.

1. The Classical RK4 Formula

The fourth-order Runge-Kutta method (often referred to simply as the Runge-Kutta method) is given by:

Butcher Tableau for Classical RK4:

Notice the weights $\vec{b} = \left[\frac{1}{6}, \frac{2}{6}, \frac{2}{6}, \frac{1}{6}\right]^T$, which correspond exactly to Simpson's $1/3$ rule weights across the interval $[t_n, t_{n+1}]$!

2. Derivation of Order Conditions via B-Series and Trees

For an explicit Runge-Kutta method to achieve order $p = 4$, the numerical expansion must match the Taylor series expansion for all rooted trees up to order 4:

1. Order 1 (1 condition): $\sum b_i = 1$

2. Order 2 (1 condition): $\sum b_i c_i = 1/2$

3. Order 3 (2 conditions):

• $\sum b_i c_i^2 = 1/3$
• $\sum b_i a_{ij} c_j = 1/6$

4. Order 4 (4 conditions):

• $\sum b_i c_i^3 = 1/4$
• $\sum b_i c_i a_{ij} c_j = 1/8$
• $\sum b_i a_{ij} c_j^2 = 1/12$
• $\sum b_i a_{ij} a_{jk} c_k = 1/24$

Together, these form a system of 8 nonlinear algebraic equations for the coefficients $\{a_{ij}, b_i, c_i\}$. The classical RK4 coefficients satisfy all 8 conditions identically, resulting in:

3. Butcher's Barrier Theorem for Higher Orders

A remarkable mathematical property discovered by J. C. Butcher is that the minimum number of stages $s(p)$ required to achieve order $p$ satisfies:

Theorem (Butcher's First Barrier): No explicit Runge-Kutta method of order $p \ge 5$ can have $s = p$ stages. Specifically, an order 5 explicit RK method requires at least $s = 6$ stages! This explains why RK4 is universally regarded as the computational sweet spot in scientific computing: it achieves 4th-order accuracy with the minimum possible number of stages ($s = 4$).