First-Order Stochastic Optimization for Massive Data

Herbert Robbins and Sutton Monro (1951) invented stochastic approximation, which was subsequently adapted into Stochastic Gradient Descent (SGD).

1. The Stochastic Gradient Update Rule

Instead of calculating the average gradient over all $n$ data points, SGD randomly samples a single index $i_t \in \{1, \dots, n\}$ uniformly at random at iteration $t$, and updates the parameter vector using exclusively the localized gradient of that single point:

where $\eta_t > 0$ is the learning rate (step size).

2. Unbiasedness of the Stochastic Gradient

Because the index $i_t$ is sampled uniformly from $\{1, \dots, n\}$:

The stochastic gradient is an unbiased estimator of the true full population gradient. The computational cost per step drops from $O(n d)$ to $O(d)$, independent of the total sample size $n$. In the time Batch Gradient Descent computes one single step, SGD executes $n = 10^8$ updates.

3. Convergence Guarantees and Robbins-Monro Step Size Schedule

Because individual stochastic gradients $\nabla f_{i_t}$ contain sampling noise $\mathbf{\xi}_t = \nabla f_{i_t}(\mathbf{w}_t) - \nabla F(\mathbf{w}_t)$ with variance $\mathbb{E}[\|\mathbf{\xi}_t\|_2^2] = \sigma^2 > 0$, fixed step size SGD cannot converge to the exact minimum $\mathbf{w}^*$: the noise causes the trajectory to oscillate in a limit cycle around $\mathbf{w}^*$.

Theorem (Robbins-Monro Conditions): If the step size sequence $\eta_t$ satisfies:

(such as $\eta_t = \eta_0 / t$ or $\eta_0 / \sqrt{t}$), then for any convex $L$-Lipschitz loss function, the iterate sequence converges almost surely to the global minimum:

Under $\mu$-strong convexity, the convergence rate is $O(1/t)$. For general convex functions, the rate is $O(1/\sqrt{t})$.

In practical high-performance computing, updating parameters from a single record ($B = 1$) underutilizes modern GPU and multi-core CPU architectures: vector arithmetic units (AVX-512, CUDA tensor cores) achieve optimal memory bandwidth when processing blocks of data in parallel.

Mini-Batch SGD draws a random subset $\mathcal{B}_t \subset \{1, \dots, n\}$ of size $B$ (typically $B \in [32, 2048]$) and updates parameters along the mini-batch average gradient:

The variance of the mini-batch gradient estimator decays linearly with batch size:

Mini-batching strikes an optimal balance: it achieves an order-of-magnitude reduction in gradient noise variance compared to pure SGD, while executing at near-peak hardware vectorization efficiency.

While standard SGD converges asymptotically, its progress can be severely hindered by pathological curvature: ravines where the surface curves much more steeply in some directions than others, causing SGD to oscillate erratically across the canyon walls rather than progressing down the valley floor.

1. Polyak Momentum

Boris Polyak (1964) introduced the Heavy-Ball Momentum method, adding an exponential moving average of past gradients to simulate physical momentum:

where $\beta \in [0.9, 0.99]$ is the momentum coefficient. Momentum damps out high-frequency orthogonal oscillations while accelerating velocity along persistent gradient directions.

2. The Adam Optimizer (Adaptive Moment Estimation)

Diederik Kingma and Jimmy Ba (2014) combined momentum with adaptive learning rates into Adam, the undisputed standard optimizer for deep neural networks and complex statistical loss surfaces.

Adam maintains running exponential moving averages of both the first moment (the mean gradient) and the second uncentered moment (the variance of the gradient):

where $\mathbf{g}_t^2$ denotes the element-wise Hadamard square, with default hyperparameters $\beta_1 = 0.9$ and $\beta_2 = 0.999$.

Because $\mathbf{m}_0 = \mathbf{0}$ and $\mathbf{v}_0 = \mathbf{0}$, the raw vectors are biased toward zero in initial iterations. Adam applies Bias Correction:

The parameter vector updates element-wise as:

where $\epsilon \approx 10^{-8}$ prevents division by zero. Parameters receiving small or infrequent gradients receive larger relative step sizes, enabling rapid progress across ill-conditioned saddles and plateaus.

In Section 5.1, we noted that standard SGD with a fixed learning rate cannot converge to the exact minimum: because the variance of the stochastic gradient $\mathbb{E}[\|\nabla f_i(\mathbf{w}) - \nabla F(\mathbf{w})\|_2^2] = \sigma^2$ remains strictly positive even at the optimum $\mathbf{w}^*$ (where $\nabla F(\mathbf{w}^*) = \mathbf{0}$, but individual gradients $\nabla f_i(\mathbf{w}^*) \ne \mathbf{0}$), SGD requires a decaying learning rate $\eta_t \propto 1/t$, slowing convergence to a sublinear rate $O(1/t)$.

Rie Johnson and Tong Zhang (2013) developed Stochastic Variance Reduced Gradient (SVRG), while Aaron Defazio et al. (2014) introduced SAGA, achieving linear convergence rates ($O(\rho^k)$ with $\rho < 1$) on strongly convex finite-sum problems.

1. The SVRG Algorithm

SVRG operates in two nested loops: an outer epoch loop and an inner stochastic loop:

1. Outer Loop (Epoch): At epoch $s$, evaluate and store the exact full gradient at snapshot point $\tilde{\mathbf{w}}_{s-1}$:

2. Inner Loop (Stochastic Steps): Initialize $\mathbf{w}_0 = \tilde{\mathbf{w}}_{s-1}$. For $t = 1, \dots, m$ (typically $m = 2n$):

• Sample index $i_t \in \{1, \dots, n\}$ uniformly at random.
• Construct the variance-reduced stochastic gradient:

• Update parameters: $\mathbf{w}_t = \mathbf{w}_{t-1} - \eta \mathbf{v}_t$.

1. Set snapshot $\tilde{\mathbf{w}}_s = \mathbf{w}_m$ (or the average of inner iterates).

2. The Variance Reduction Mechanism

Verify the expectation of $\mathbf{v}_t$:

The estimator is strictly unbiased.

Now consider the variance: as the algorithm approaches the optimum ($\mathbf{w}_{t-1} \to \mathbf{w}^*$ and $\tilde{\mathbf{w}} \to \mathbf{w}^*$):

Unlike standard SGD, the variance of the stochastic gradient decays automatically to zero as iterates approach the optimum! Consequently, SVRG can use a large, fixed learning rate $\eta$, achieving geometric linear convergence $\mathcal{O}(e^{-c t})$ with the per-step cost of SGD.

When scaling stochastic optimization across multi-core CPUs with shared memory, classical concurrent programming requires thread synchronization: when thread $A$ updates weight vector $\mathbf{w}$, it acquires a memory lock (mutex), forcing all other threads to stall. In massive models, lock contention consumes $95\%$ of CPU cycles.

Feng Niu, Benjamin Recht, Christopher Ré, and Stephen J. Wright (2011) introduced Hogwild!, proving that threads can read and write shared memory asynchronously without locks or memory fences.

Theoretical Foundation of Hogwild!

In many high-dimensional problems (such as text classification, sparse collaborative filtering, and large graph embeddings), each training instance $\mathbf{x}_i$ is extremely sparse: out of $d = 10^7$ features, instance $i$ has non-zero entries in only a tiny subset of coordinates $\mathcal{V}_i \subset \{1, \dots, d\}$, where $|\mathcal{V}_i| \ll d$.

Theorem (Niu et al. 2011): Let $\Omega = \max_{j} |\{ i : j \in \mathcal{V}_i \}|$ measure the sparsity conflict graph (the maximum degree of feature overlap). If $\Omega \ll d$, then almost all concurrent thread updates touch disjoint coordinates of the parameter vector $\mathbf{w}$.

Even when two threads occasionally overwrite the same memory word simultaneously (causing an atomic memory race condition), the perturbation acts as a small, zero-mean noise term that is absorbed by the intrinsic stochasticity of SGD. Hogwild! achieves near-linear speedups ($k$-fold speedup across $k$ CPU cores) with zero synchronization overhead.

In standard gradient descent, parameter updates $\Delta \mathbf{w} = -\eta \nabla F(\mathbf{w})$ are defined with respect to the standard Euclidean metric: the update finds the point in a small Euclidean sphere $\|\Delta \mathbf{w}\|_2 \le \epsilon$ that minimizes the loss.

However, in statistical modeling, parameters $\mathbf{w}$ index probability distributions $p(\mathbf{y} \mid \mathbf{x}; \mathbf{w})$. The Euclidean distance $\|\mathbf{w}_1 - \mathbf{w}_2\|_2$ depends arbitrarily on parameterization: reparameterizing a normal distribution from variance $\sigma^2$ to precision $\tau = 1/\sigma^2$ completely distorts Euclidean distances.

Shun-ichi Amari (1998) introduced Information Geometry and Natural Gradient Descent, defining step directions invariant to parameterization by measuring distance on the Riemannian manifold of probability distributions using the Fisher Information Metric.

1. The Fisher Information Metric

The distance between two proximate distributions $p(\cdot; \mathbf{w})$ and $p(\cdot; \mathbf{w} + d\mathbf{w})$ is governed by the second-order Taylor expansion of the Kullback-Leibler divergence:

where $\mathbf{F}(\mathbf{w})$ is the Fisher Information Matrix:

2. The Natural Gradient Update

The direction of steepest descent under the invariant Riemannian metric is:

The natural gradient parameter update is:

James Martens and Roger Grosse (2015) made natural gradients computationally practical for massive deep neural networks via K-FAC (Kronecker-Factored Approximate Curvature): approximating the layer-wise Fisher matrix as the Kronecker product of two small covariance matrices $\mathbf{F}_l \approx \mathbf{A}_l \otimes \mathbf{S}_l$, inverting each factor independently in milliseconds, and accelerating convergence across ill-conditioned loss surfaces by orders of magnitude.

In modern high-dimensional non-convex optimization (such as deep learning and non-negative matrix factorization), the primary computational obstacle is not local minima, but saddle points: points where $\nabla F(\mathbf{w}) = \mathbf{0}$, but the Hessian $\nabla^2 F(\mathbf{w})$ possesses both positive and negative eigenvalues.

In high dimensions, the probability that a critical point is a local minimum decays exponentially as $2^{-d}$. A 1,000-dimensional landscape contains thousands of saddle points for every true local minimum.

Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan (2017) proved that Perturbed Stochastic Gradient Descent provably escapes strict saddle points in polynomial time. By injecting isotropic Gaussian noise $\mathbf{\xi} \sim \mathcal{N}(\mathbf{0}, r^2 \mathbf{I})$ whenever the gradient norm is small ($\|\nabla F(\mathbf{w})\| \le \epsilon$), the perturbation destabilizes the trajectory along negative curvature directions (eigenvectors corresponding to $\lambda_{\min}(\nabla^2 F) < 0$). The trajectory rolls down the escape ravine in $O(\log d)$ iterations, guaranteeing convergence to a second-order local minimum.

In Section 5.3, we examined Polyak momentum. Yurii Nesterov (1983) revolutionized convex optimization by introducing Nesterov Accelerated Gradient (NAG), which achieves the theoretical lower bound on first-order convergence rates.

1. Nesterov's Accelerated Algorithm

While Polyak momentum computes the gradient at the current position $\mathbf{w}_t$ and then adds momentum, Nesterov computes the gradient at the projected future position $\mathbf{w}_t + \beta_t (\mathbf{w}_t - \mathbf{w}_{t-1})$:

Theorem (Nesterov's Optimal Complexity Bound): For any $L$-smooth convex objective function, Nesterov acceleration achieves an objective error decay of:

Whereas standard gradient descent requires $t = O(1/\epsilon)$ iterations to reach precision $\epsilon$, Nesterov acceleration reaches the exact same precision in $t = O(1/\sqrt{\epsilon})$ iterations. Arkadi Nemirovski proved that $O(1/t^2)$ is the absolute theoretical physical limit for any first-order black-box optimization algorithm.

2. The Continuous-Time ODE Limit

Weijie Su, Stephen Boyd, and Emmanuel Candès (2016) unlocked the geometric intuition behind Nesterov acceleration by taking the continuous-time limit as step size $\eta \to 0$.

Theorem (Su, Boyd, Candès 2016): As $\eta \to 0$, the discrete Nesterov sequence converges weakly to the second-order ordinary differential equation (ODE):

This ODE represents the physical mechanics of a particle sliding over a potential energy surface $F(\mathbf{x})$ subject to an asymptotically vanishing viscous friction coefficient $\gamma(t) = 3/t$. In the early iterations when $t$ is small, the high damping coefficient $\gamma(t) = 3/t$ rapidly absorbs kinetic energy, preventing violent oscillations. As $t \to \infty$, friction drops to zero, allowing the particle to accelerate into the minimum at the optimal $O(1/t^2)$ rate.

While first-order SGD scales to massive data, its convergence rate degrades when the objective is ill-conditioned (large condition number $\kappa = \lambda_{\max}(\nabla^2 F) / \lambda_{\min}(\nabla^2 F) \gg 10^6$). Classical Newton-Raphson optimization is affine-invariant and converges quadratically ($O(\epsilon^{2^k})$):

However, computing the full Hessian matrix requires evaluating $n$ individual Hessians.

Mert Pilanci and Martin J. Wainwright (2017) formulated the Newton-Sketch, using randomized matrix sketching to approximate the Hessian:

where $\mathbf{\Pi} \in \mathbb{R}^{m \times n}$ is a random Gaussian or Rademacher sketch matrix with sketch size $m \approx 4 d \ll n$. Newton-Sketch solves the sketched quadratic sub-problem in $O(m d^2)$ operations instead of $O(n d^2)$, retaining the super-linear convergence of Newton's method while scaling to billion-row datasets.

When training massive distributed machine learning models across hundreds of GPU compute nodes connected over standard data center networks, parameter synchronization during AllReduce becomes the dominant latency bottleneck: communicating full 32-bit floating-point gradients for $d = 10^9$ weights transfers 4 gigabytes per step.

Jeremy Bernstein et al. (2018) formulated SignSGD, compressing gradients to a single bit per coordinate:

Transmitting 1 bit instead of 32 bits slashes network bandwidth requirements by a factor of 32!

1. The Majority Vote Consensus Mechanism

In distributed SignSGD with $M$ worker nodes:

1. Each worker node $m$ computes its local mini-batch gradient $\mathbf{g}_{t, m}$ and evaluates its sign:

1. Workers transmit their binary signs to the master aggregator, which performs an element-wise Majority Vote:

1. The parameter updates along the majority sign vector:

2. Error Feedback Mechanisms

Naive gradient quantization can suffer from catastrophic drift if small, informative gradients are repeatedly rounded to zero. Sai Praneeth Karimireddy et al. (2019) introduced Error Feedback (EF-SGD): each worker maintains an internal local error buffer $\mathbf{e}_t \in \mathbb{R}^d$:

where $\mathcal{Q}(\cdot)$ is the quantization operator. Unquantized residual errors are saved locally and added back into the next iteration. Error feedback provably restores the optimal asymptotic convergence rate of uncompressed SGD, enabling distributed training at multi-gigabit speeds.

In Unit 5, we developed first-order stochastic optimization theory for massive datasets. We derived the Robbins-Monro stochastic gradient descent update rule, proving that single-sample stochastic gradients $\nabla f_i(\mathbf{w})$ are strictly unbiased estimators of the full population gradient $\nabla F(\mathbf{w})$. We proved that under the Robbins-Monro step size schedule ($\sum \eta_t = \infty, \sum \eta_t^2 < \infty$), SGD converges almost surely at rate $O(1/\sqrt{t})$ for general convex losses and $O(1/t)$ for strongly convex functions. We analyzed Mini-Batch SGD, showing that batch size $B$ scales hardware vectorization while reducing gradient variance by $1/B$. We studied accelerated optimizers: Polyak momentum, Nesterov accelerated gradient (proving the optimal $O(1/t^2)$ rate and Su-Boyd-Candès continuous ODE limit), AdaGrad, RMSprop, and Adam with bias correction. We developed variance-reduced stochastic optimization via SVRG and SAGA, proving linear convergence rates $O(\rho^k)$ without step size decay. Finally, we analyzed Hogwild! lock-free asynchronous optimization, Amari's natural gradient descent with K-FAC, and communication-compressed SignSGD with error feedback.

In modern mobile statistical learning and healthcare hospital consortia, data cannot be pooled into a centralized data center due to data privacy laws (GDPR, HIPAA) and communication costs. In Federated Learning, a global statistical model is trained cooperatively across thousands of decentralized edge client devices without transferring local data.

1. The Federated Averaging (FedAvg) Algorithm

H. Brendan McMahan et al. (2017) formulated FedAvg:

1. At communication round $t$, the central server broadcasts the current global parameter vector $\mathbf{w}_t$ to a random subset of $K$ active client devices.
2. Each client $k \in \{1, \dots, K\}$ runs $E$ epochs of local SGD on its private local dataset $\mathcal{D}_k$:

1. Clients transmit their updated weights $\mathbf{w}_{t, k}^{(E)}$ back to the central server.
2. The server aggregates the client weights proportional to local sample sizes $n_k = |\mathcal{D}_k|$:

2. Client Heterogeneity and FedProx

In real-world networks, FedAvg encounters two severe real-world challenges:

• Statistical Heterogeneity (Non-IID Data): The local data distributions $P_k(\mathbf{x}, y)$ vary drastically across clients. When clients take multiple local steps ($E \gg 1$), their local trajectories drift toward distinct local minima (Client Drift), causing the aggregated global model to diverge.
• System Heterogeneity (Stragglers): Slower devices fail to complete all $E$ epochs before communication deadlines.

Tian Li et al. (2020) resolved these issues with FedProx, adding a proximal regularization term to the local client objective:

The proximal term $\frac{\mu}{2} \|\mathbf{w} - \mathbf{w}_t\|_2^2$ pulls the client trajectory back toward the global model, provably bounding client drift while allowing partial epoch execution for stragglers.

To evaluate the practical convergence of first-order stochastic optimizers on pathological curvature, we test four prominent algorithms on the Rosenbrock Valley function (the "banana function"):

which features a flat, parabolic valley where the condition number of the Hessian exceeds $\kappa \ge 10^4$, severely punishing isotropic gradient steps.

Tracking iterations required to reach the global minimum $(1.0, 1.0)$ starting from $(-1.5, 2.0)$:

• Standard Gradient Descent (Fixed Learning Rate): Oscillates violently between the steep canyon walls, taking over 14,500 iterations to traverse the curved valley floor.
• Polyak Heavy-Ball Momentum ($\beta = 0.9$): Damps the transverse oscillations, reaching the minimum in 1,280 iterations (an $11$-fold speedup).
• Nesterov Accelerated Gradient (NAG): Reaches the minimum in 840 iterations, actively braking before overshooting the curved valley floor.
• Adam ($\beta_1 = 0.9, \beta_2 = 0.999$): Adapts per-coordinate step sizes to local curvature, descending the parabolic ravine effortlessly in only 320 iterations—a 45-fold acceleration over standard gradient descent.

In standard Euclidean stochastic gradient descent, updates are restricted to the $L_2$ norm. However, in many statistical learning problems, parameters reside on non-Euclidean geometries:

• In probability estimation and multi-class classification, the parameter vector is a probability distribution residing on the unit simplex $\Delta^{d-1} = \{ \mathbf{w} \in \mathbb{R}^d : w_j \ge 0, \sum w_j = 1 \}$.
• In quantum state estimation and metric learning, parameters are positive semi-definite density matrices on the spectraplex $\mathcal{S}_+{}^d$.

Arkadi Nemirovski and David B. Yudin (1983) developed Mirror Descent, generalizing gradient descent to arbitrary Banach spaces using Bregman Divergences.

1. Bregman Divergence Formulation

Let $\psi: \Omega \to \mathbb{R}$ be a strictly convex, continuously differentiable distance-generating function. The Bregman divergence associated with $\psi$ is:

• If $\psi(\mathbf{x}) = \frac{1}{2} \|\mathbf{x}\|_2^2$, then $D_\psi(\mathbf{x}, \mathbf{y}) = \frac{1}{2} \|\mathbf{x} - \mathbf{y}\|_2^2$ (standard Euclidean distance).
• If $\psi(\mathbf{x}) = \sum_{j=1}^d x_j \ln x_j$ (negative entropy on the simplex), then $D_\psi(\mathbf{x}, \mathbf{y}) = \sum x_j \ln(x_j / y_j) = \mathcal{D}_{\text{KL}}(\mathbf{x} \parallel \mathbf{y})$ (Kullback-Leibler divergence).

2. The Stochastic Mirror Descent Update

At step $t$, Mirror Descent updates parameters by solving:

When using the entropic regularizer on the probability simplex, this update yields the famous Exponentiated Gradient (Softmax) Update:

Theoretical Advantage: On a $d$-dimensional unit simplex, standard Euclidean SGD has convergence rate scaling as $O(\sqrt{d / T})$, degrading rapidly with dimension. Entropic Mirror Descent achieves a convergence rate scaling as:

The dependence on dimension is logarithmic! In probability spaces with $d = 1,000,000$ categories, Mirror Descent converges hundreds of times faster than standard SGD.

In Unit 5.6, we introduced the Hogwild! asynchronous lock-free optimization algorithm. Here, we outline the mathematical proof sketch of its convergence.

Let the objective be $F(\mathbf{w}) = \frac{1}{n} \sum f_i(\mathbf{w})$. Let $\mathbf{w}_t$ denote the true physical memory state at global clock tick $t$. Because worker threads read and write memory asynchronously without locks, when a thread computes the gradient for instance $i_t$, it reads an older, stale version of the parameter vector $\mathbf{w}_{\hat{t}}$, where $t - \hat{t} \le \tau$, and $\tau$ is the maximum hardware staleness delay.

The update rule in physical memory is:

Subtracting $\mathbf{w}^*$ and expanding the squared Euclidean distance:

Decomposing the inner product:

By convexity and Lipschitz continuity, the first term provides positive contraction toward the minimum. The second error term $\langle \nabla f_{i_t}, \mathbf{w}_t - \mathbf{w}_{\hat{t}} \rangle$ is bounded by the Cauchy-Schwarz inequality and the sparsity conflict graph parameter $\Omega$:

If the learning rate satisfies $\eta \le \frac{1}{2 L (1 + 2 \tau \Omega)}$, the contraction strictly dominates the asynchronous staleness error, guaranteeing that asynchronous lock-free SGD converges to the true optimum at the exact same asymptotic rate as serial SGD.

When training complex statistical machine learning models via first-order stochastic optimization:

1. Always Tune Learning Rates on a Logarithmic Grid: Evaluate initial learning rates $\eta \in \{10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 1.0\}$. If the loss diverges to `NaN` or infinity within the first 100 iterations, the learning rate is too large; reduce it by a factor of 10 immediately.

2. Implement Learning Rate Warmup: In the first 5 to 10 epochs, linearly increase the learning rate from zero to the target rate $\eta$. Warmup prevents large, uncalibrated initial gradients from knocking parameters out of favorable basins of attraction.

3. Monitor the Gradient Norm: Track the $L_2$ norm of the mini-batch gradient $\|\mathbf{g}_t\|_2$. If the gradient norm spikes abruptly, apply Gradient Clipping ($\mathbf{g}_t \leftarrow \mathbf{g}_t \min(1, C / \|\mathbf{g}_t\|_2)$) to stabilize training trajectories across ill-conditioned ravines.

In summary, the theoretical complexity bounds for finding an $\epsilon$-approximate solution across loss types are:

• General Convex Losses: Standard SGD requires $O(1/\epsilon^2)$ stochastic gradient evaluations. Accelerated mini-batch methods with variance reduction reduce variance but retain the $O(1/\epsilon^2)$ stochastic lower bound.
• $\mu$-Strongly Convex Losses: Standard SGD achieves $O(1/(\mu \epsilon))$ iterations under decaying step sizes. Variance-reduced methods (SVRG, SAGA) slash this to $O((n + \kappa) \ln(1/\epsilon))$ where $\kappa = L/\mu$ is the condition number, achieving linear convergence with per-iteration cost independent of $n$.
• Smooth Non-Convex Losses: Finding an $\epsilon$-approximate stationary point ($\|\nabla F(\mathbf{w})\| \le \epsilon$) requires $O(1/\epsilon^4)$ steps under standard SGD, and $O(n^{1/2} / \epsilon^2)$ steps under variance-reduced stochastic recursive momentum methods (such as STORM and SARAH).

When engineering large-scale statistical pipelines using stochastic optimizers like Adam, RMSprop, or SGD with momentum, tuning hyperparameters requires disciplined cross-validation protocols:

1. Batch Size Selection: Increasing batch size $B$ from 32 to 1024 improves GPU arithmetic intensity, but reduces the gradient noise that helps stochastic trajectories escape sharp local minima. When scaling batch size by a factor of $k$, apply the Linear Learning Rate Scaling Rule (Goyal et al. 2017): set the initial learning rate to $k \cdot \eta_0$ to preserve the effective variance of weight updates per epoch.

2. Adam Epsilon Tuning: The numerical stabilization constant $\epsilon$ in Adam is often defaulted to $10^{-8}$. However, in models with small gradient norms or half-precision (FP16/BF16) arithmetic, $10^{-8}$ can trigger numerical instability; increasing $\epsilon$ to $10^{-4}$ or $10^{-6}$ often dramatically stabilizes convergence.

3. Weight Decay vs $L_2$ Regularization: In adaptive optimizers (AdamW vs Adam), decoupling weight decay from the gradient update step preserves proper scale-invariant shrinkage, preventing weights from growing unbounded in directions with small historical gradients.