Distributed Computing Foundations: MapReduce, Hadoop & Spark for Statisticians

High-performance computing historically relied on Symmetric Multiprocessing (SMP) and Non-Uniform Memory Access (NUMA) systems, where multiple CPUs share a common physical address space. However, shared-memory systems are physically limited in scalability (rarely exceeding 128 cores) and are vulnerable to bus lock contention.

Modern big data frameworks operate on the Shared-Nothing Architecture:

• The computing cluster consists of $M$ independent compute nodes connected via standard Ethernet switches.
• Each node possesses its own private CPU, private memory (RAM), and local disk storage.
• Nodes share zero physical memory. All coordination occurs via message passing and network data transfer.

Jeffrey Dean and Sanjay Ghemawat (2004) at Google formulated the MapReduce programming model to abstract distributed data processing, implemented in open-source by Apache Hadoop.

1. The Map and Reduce Primitives

MapReduce partitions computation into three strictly choreographed phases:

1. Map Phase: Operates on input key-value pairs $(k_1, v_1)$ independently and in parallel across cluster nodes, emitting intermediate key-value pairs:

2. Shuffle and Sort Phase: The distributed runtime automatically collects all intermediate values sharing the identical key $k_2$ across all nodes, grouping them into a single list $(k_2, \text{list}(v_2))$ and routing them to the designated reducer node.

3. Reduce Phase: Aggregates the values associated with key $k_2$:

2. Parallel Evaluation of Sufficient Statistics

Let $\mathbf{x}_1, \dots, \mathbf{x}_N$ be $N$ independent observations distributed across $M$ worker nodes, where $N \gg 10^9$. A statistic $T(\mathbf{X})$ is defined as associative and commutative if it can be decomposed into localized partition evaluations:

Examples of MapReduce decompositions in statistics:

1. Sample Mean:

• Mapper: on node $m$, compute local count $n_m = |S_m|$ and local sum $s_m = \sum_{i \in S_m} x_i$. Emit key-value pair ("mean", (n_m, s_m)).
• Reducer: sum local counts $N = \sum n_m$ and local sums $S = \sum s_m$. Emit $\bar{x} = S / N$.

Network transfer is $O(M)$ floating-point numbers instead of $O(N)$ data records.

2. Gram Matrix $\mathbf{X}^T \mathbf{X}$:

• Mapper: on node $m$, compute the local $p \times p$ matrix $\mathbf{G}_m = \mathbf{X}^{(m)T} \mathbf{X}^{(m)}$. Emit ("gram", G_m).
• Reducer: sum $\mathbf{X}^T \mathbf{X} = \sum_{m=1}^M \mathbf{G}_m$.

Once the $p \times p$ Gram matrix and $p \times 1$ vector $\mathbf{X}^T \mathbf{y}$ are aggregated, the master node solves the normal equations $(\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$ in $O(p^3)$ time without ever storing the full dataset in memory.

While Hadoop MapReduce revolutionized batch data processing, it suffered from a fatal performance flaw in statistical applications: MapReduce writes all intermediate outputs to physical disks at the end of every Map and Reduce cycle. Iterative algorithms—such as Gradient Descent, Expectation-Maximization (EM), and MCMC—require dozens of sequential iterations, incurring catastrophic disk I/O bottlenecks.

Matei Zaharia et al. (2010) created Apache Spark to enable high-speed in-memory distributed data analysis.

1. Resilient Distributed Datasets (RDDs)

An RDD is an immutable, lazily evaluated, fault-tolerant distributed collection of data records partitioned across cluster worker nodes. RDDs are governed by two distinct operations:

• Transformations (Lazy): Construct a new RDD from an existing RDD (e.g., map, filter, flatMap, groupByKey). Transformations do not execute immediately; Spark constructs a Directed Acyclic Graph (DAG) representing the Lineage Graph.
• Actions (Eager): Trigger cluster execution and return a result to the driver program or write to storage (e.g., count, collect, reduce, take).
• Fault Tolerance via Lineage: If a cluster worker node crashes mid-computation, Spark does not restore from periodic disk checkpoints. Instead, it recomputes only the lost partition by replaying the mathematical transformations along the RDD lineage graph.

2. In-Memory Caching for Iterative Machine Learning

By invoking rdd.cache() or rdd.persist(StorageLevel.MEMORY_ONLY), the dataset partitions are pinned directly inside worker node RAM. In subsequent iterations of an optimization algorithm:

the gradient evaluation streams through cached RAM at memory-bus speeds (tens of gigabytes per second), executing machine learning loops 100 times faster than disk-backed Hadoop.

In distributed statistical learning, computational complexity $T_{\text{comp}}$ measures the total floating-point operations executed across all processors, while communication complexity $T_{\text{comm}}$ measures the total volume of network bits transmitted and the latency of network synchronization barriers.

Let $P$ be the number of worker nodes, $N$ the total sample size, and $p$ the parameter dimension:

• Local Gradient Evaluation: $O(N p / P)$ computational operations per node.
• AllReduce Parameter Broadcast: Transmitting a $p$-dimensional parameter vector to all $P$ workers requires $O(p \log P)$ communication volume.
• If $p$ is massive ($p = 10^7$) or network bandwidth is congested, worker nodes spend $90\%$ of their time waiting for network packet synchronization (communication-bound execution).

To minimize communication overhead, statisticians deploy:

1. Local SGD / Federated Averaging: Workers execute $K$ local gradient steps independently before synchronizing parameter averages across the network.

2. Gradient Compression and Quantization: Transmitting 1-bit or 8-bit quantized gradients (e.g., SignSGD) instead of 64-bit double-precision floats.

When data are partitioned across $K$ worker machines, running iterative gradient algorithms requiring frequent communication is often impractical over high-latency networks.

A natural distributed strategy is One-Shot Parameter Averaging (Divide-and-Conquer):

1. Distribute $N$ observations across $K$ nodes, so each node holds a local sample of size $n = N/K$.
2. Each worker computes its local M-estimator independently:

1. Transmit the $K$ local vectors to a central coordinator and compute the simple average:

Yuchen Zhang, John C. Duchi, and Martin J. Wainwright (2013) established the exact statistical efficiency bounds of this one-shot scheme.

Asymptotic Mean Squared Error Expansion

Let $\boldsymbol{\theta}^*$ be the true parameter, and let $\ell$ be a smooth loss function with Fisher information matrix $\mathbf{I}(\boldsymbol{\theta}^*)$. The local M-estimator on worker $k$ admits a second-order Taylor expansion:

where $\mathbf{b}(\boldsymbol{\theta}^*)$ is the second-order bias term originating from the non-linearity of the score function.

Averaging across the $K$ workers:

The Mean Squared Error (MSE) decomposes into variance and squared bias:

Theorem (Zhang, Duchi, Wainwright 2013): If the number of worker nodes satisfies:

(meaning local sample size $n = N/K \ge \sqrt{N}$), then the squared bias term $O(n^{-4})$ is asymptotically dominated by the variance $O(1/N)$. Under this condition, one-shot parameter averaging achieves the exact minimax optimal statistical efficiency of a centralized supercomputer while transmitting data across the network exactly once!

In big data analytics, the physical layout of statistical tables in memory and on disk dictates computational performance. Historically, database systems stored data in Row-Oriented formats (e.g., CSV, JSON, relational database rows): Record 1: [ID, Age, Income, Diagnosis] Record 2: [ID, Age, Income, Diagnosis]

1. The Columnar Revolution: Apache Parquet

In statistical analysis, queries rarely access all columns: an econometric regression model $Y \sim X_1 + X_2$ extracts 3 columns out of a dataset containing 2,000 columns. In a row-oriented file, the computer must read every byte of all 2,000 columns from disk, discarding $99.8\%$ of data in memory.

Apache Parquet stores data in Columnar Format: Age: [25, 42, 67, ...] Income: [55000, 82000, 110000, ...] Disease: [0, 1, 0, ...]

Advantages for statisticians:

1. Projection Pushdown (Column Pruning): Disk reads fetch only the exact bytes corresponding to variables included in the model, slashing disk I/O by $95\%$ to $99\%$.

2. Homogeneous Data Compression: Because all values in a column share the identical data type, specialized columnar encoding algorithms (Run-Length Encoding - RLE, Dictionary Encoding, Delta Encoding) achieve $10\times$ higher compression ratios than row-based formats.

3. Predicate Pushdown: Parquet files record the minimum and maximum values (`min/max` metadata) for every block of data. If a query filters `WHERE Age > 65`, blocks where `max(Age) < 65` are skipped entirely without reading a single record.

2. Zero-Copy In-Memory Interoperability: Apache Arrow

Even when data are stored efficiently on disk, moving data between different statistical environments (e.g., passing a Spark DataFrame to Python Pandas or R) historically required expensive serialization and deserialization (copying memory and converting types), consuming $80\%$ of runtime.

Apache Arrow standardizes an in-memory columnar memory specification across languages (C++, Python, R, Java, Rust). Because all engines adhere to the identical memory layout:

• Data are shared via shared-memory pointers (Zero-Copy Data Transfer).
• Computational routines execute SIMD (Single Instruction Multiple Data) vector instructions directly over contiguous memory arrays, processing billions of rows per second.

In distributed statistical estimation, data are partitioned across $K$ worker nodes. Stephen Boyd et al. (2011) popularized the Alternating Direction Method of Multipliers (ADMM), providing a mathematically robust framework for solving distributed convex optimization problems with consensus constraints.

1. The Global Consensus Problem

Consider solving the distributed empirical risk minimization problem across $K$ nodes:

where $f_k(\mathbf{w})$ is the local loss function on worker $k$, and $g(\mathbf{w})$ is a global regularization term (such as the $L_1$ penalty $g(\mathbf{w}) = \lambda \|\mathbf{w}\|_1$).

We introduce local worker copies $\mathbf{w}_1, \dots, \mathbf{w}_K \in \mathbb{R}^d$ and an auxiliary master consensus variable $\mathbf{z} \in \mathbb{R}^d$, reformulating the objective as:

2. The Augmented Lagrangian and ADMM Updates

The augmented Lagrangian with penalty parameter $\rho > 0$ and dual multipliers $\mathbf{u}_k \in \mathbb{R}^d$ is:

ADMM updates execute in three distributed stages:

1. Parallel Worker Step: Each worker $k \in \{1, \dots, K\}$ minimizes its local augmented loss independently:

Worker updates require zero network communication and execute in parallel across nodes.

2. Master Consensus Step: The central coordinator gathers local vectors $\mathbf{w}_k^{(t+1)}$, computes the sample average $\bar{\mathbf{w}}^{(t+1)} = \frac{1}{K} \sum \mathbf{w}_k^{(t+1)}$, and solves the proximal step:

When $g(\mathbf{z}) = \lambda \|\mathbf{z}\|_1$, this update is computed in closed form using the soft-thresholding operator $\mathcal{S}(\cdot, \lambda / (K \rho))$.

3. Dual Multiplier Update: Each worker updates its dual variable:

ADMM converges to the exact global optimum under arbitrary data partitioning, handling non-smooth regularizers and ill-conditioned losses with guaranteed primal and dual residual decay.

When fitting modern deep recommendation systems, natural language embeddings, or massive search click-through-rate models, the parameter vector $\mathbf{w}$ contains billions of weights (e.g., $d \ge 10^{11}$, requiring terabytes of storage for parameters alone). In this regime, the parameter vector cannot fit into the memory of any single master coordinator node.

Mu Li et al. (2014) introduced the Parameter Server architecture:

• Nodes are partitioned into two groups: Worker Nodes (which store data shards and compute gradients) and Server Nodes (which partition and maintain the global parameter vector $\mathbf{w}$).
• The global parameter space is partitioned across server nodes using distributed consistent hashing.
• Workers transmit and receive parameters asynchronously using two primitives:
• Push(keys, gradients): Worker sends sparse gradients for the specific features present in its mini-batch to the designated server node.
• Pull(keys): Worker fetches the current parameter weights for its active mini-batch features.

By decoupling model storage from data processing and supporting bounded asynchronous updates (Stale Synchronous Parallelism - SSP), parameter servers scale statistical learning models to hundreds of billions of parameters across clusters of thousands of machines.

When training statistical machine learning models across distributed consumer devices (such as smart phones or hospital patient registries in Federated Learning), transmitting raw individual data or exact gradients exposes sensitive private information: an adversarial observer intercepting gradient transmissions can reconstruct original text messages or patient health records using gradient inversion attacks.

Cynthia Dwork (2006) formulated Differential Privacy, adapted into Local Differential Privacy (LDP) by John Duchi, Michael I. Jordan, and Martin J. Wainwright (2013).

1. Mathematical Definition of Local Differential Privacy

A randomized mechanism $\mathcal{M}: \mathcal{X} \to \mathcal{Z}$ satisfies $\epsilon$-Local Differential Privacy if for any two input values $x, x' \in \mathcal{X}$ and any measurable subset of outputs $S \subseteq \mathcal{Z}$:

where $\epsilon > 0$ is the privacy budget. A smaller $\epsilon$ guarantees stronger privacy protection, ensuring that an attacker observing output $\mathcal{M}(x)$ cannot distinguish whether the input was $x$ or $x'$.

2. The Laplace and Gaussian Perturbation Mechanisms

In distributed statistical estimation of a vector-valued statistic $\boldsymbol{\theta}(\mathbf{X}) \in \mathbb{R}^d$, workers inject calibrated random noise before transmitting updates:

1. The Laplace Mechanism:

where $\Delta_1 = \max_{\mathbf{x}, \mathbf{x}'} \|\boldsymbol{\theta}(\mathbf{x}) - \boldsymbol{\theta}(\mathbf{x}')\|_1$ is the $L_1$ global sensitivity.

2. The Gaussian Mechanism:

satisfying $(\epsilon, \delta)$-differential privacy, where $\Delta_2$ is the $L_2$ global sensitivity.

Statistical Minimax Privacy Trade-Off: Duchi, Jordan, and Wainwright proved that under $\epsilon$-LDP, the minimax estimation rate for estimating a $d$-dimensional mean vector drops from the classical rate $O(d/n)$ to:

Privacy imposes an unavoidable statistical cost: achieving high privacy ($\epsilon \ll 1$) requires scaling sample size $n$ by $1/\epsilon^2$.

In distributed SQL and DataFrame engines (such as Spark SQL, Trino, and Google BigQuery), computing a database join between two distributed tables $\mathbf{A} \bowtie \mathbf{B}$ on a common key is the most communication-intensive operation in big data systems.

Engines deploy two fundamentally distinct join strategies:

1. Shuffle Hash Join: Both tables $\mathbf{A}$ and $\mathbf{B}$ are repartitioned across all $P$ cluster nodes by computing a hash of the join key $h(\text{key}) \pmod P$. Records with identical keys are transmitted over the physical cluster network to the same destination node, which performs a local hash join.

Network Cost: $O(|\mathbf{A}| + |\mathbf{B}|)$ full network transmission volume. If both tables contain terabytes of data, shuffling saturates the network switches.

2. Broadcast Hash Join (Map-Side Join): When one table is small (e.g., table $\mathbf{B}$ is a dimension table of 50 MB, while table $\mathbf{A}$ is a fact table of 2 TB), the engine avoids shuffling table $\mathbf{A}$ entirely. The master node broadcasts a complete copy of table $\mathbf{B}$ to the local RAM of every worker node. Each worker performs the join locally against its existing partition of table $\mathbf{A}$.

Network Cost: $O(P \cdot |\mathbf{B}|)$ network transmission. Zero shuffle traffic for table $\mathbf{A}$.

Catalyst query optimizers inspect table statistics automatically: if a table size falls below spark.sql.autoBroadcastJoinThreshold (default 10 MB), it automatically promotes the query to a Broadcast Hash Join, accelerating distributed queries by factors of 10 to 100.

In modern network statistics (such as social networks, citation graphs, protein interaction networks, and financial transaction money-laundering detection), data are represented as massive directed graphs $G = (V, E)$ containing billions of vertices and edges. Expressing iterative graph algorithms (such as PageRank, connected components, and label propagation) in classical MapReduce requires multiple full-table shuffles per iteration, causing crippling network overhead.

Leslie Valiant (1990) formulated the Bulk Synchronous Parallel (BSP) model, adapted by Grzegorz Malewicz et al. (2010) at Google into Pregel, and implemented in Apache Spark as GraphX.

The "Think Like a Vertex" Programming Model

Pregel organizes distributed graph computation into a sequence of synchronized Supersteps:

1. Within a superstep $S$, each active vertex $v \in V$ executes a user-defined function in parallel:

• Reads incoming messages sent to $v$ in superstep $S-1$.
• Updates its internal vertex state (e.g., current PageRank value).
• Emits outgoing messages along its adjacent edges $e = (v, u)$ to neighboring vertices.

1. At the end of the superstep, a global synchronization barrier is executed, and messages are routed across the cluster network.
2. If a vertex has completed its updates, it votes to halt. Computation terminates when all vertices are inactive.

In PageRank calculation:

GraphX keeps graph topology pinned in cluster worker RAM, exchanging only lightweight scalar messages across adjacent vertices and computing PageRank on graphs with 10 billion edges in minutes.

In the database foundations of modern data science, distributed query processing is transitioning away from the traditional Volcano Iterator Model (tuple-at-a-time processing, Graefe 1994) toward Vectorized Execution Engines (e.g., Velox, DuckDB, Apache DataFusion).

Volcano vs. Vectorized Engine Comparison

• Volcano Iterator Model: The execution engine processes queries one row at a time: next() returns a single record tuple. Every single field access triggers a virtual function call, incurring severe CPU instruction cache misses and branch mispredictions.
• Vectorized Execution (MonetDB/X100, Boncz et al. 2005): The primitive execution operator next() processes a contiguous vector batch of $N \approx 1024$ values from a single column at a time:
• Vector batches fit entirely inside the fast L1 CPU data cache.
• Loops contain zero dynamic branches or function calls.
• Compilers automatically emit AVX-512 and Neon SIMD (Single Instruction Multiple Data) hardware vector instructions, evaluating operations (e.g., $X_j > 50$) across 16 double-precision floats in a single CPU clock cycle.

Vectorized engines achieve order-of-magnitude speedups over classical distributed SQL engines, delivering real-time interactive exploratory statistics on multi-terabyte datasets.

In Unit 4, we analyzed the computational and systems architectures underpinning distributed statistics for massive data. We examined the shared-nothing cluster model and formulated the MapReduce computing paradigm, showing how associative and commutative sufficient statistics (sample means, contingency tables, Gram matrices $\mathbf{X}^T \mathbf{X}$) can be aggregated in a single pass with $O(M)$ network communication. We detailed Apache Spark's in-memory computing architecture, Resilient Distributed Datasets (RDDs), lineage graphs, and memory persistence mechanisms that execute machine learning loops 100 times faster than disk-backed Hadoop. We analyzed the communication complexity vs. computational complexity trade-off, examining local SGD and federated averaging. We proved Zhang, Duchi, and Wainwright's statistical efficiency theorem for one-shot parameter averaging, showing that when $K \le \sqrt{N}$, distributed averaging achieves optimal centralized minimax efficiency. Finally, we studied columnar formats (Apache Parquet and Apache Arrow), ADMM distributed consensus optimization, parameter servers for massive embedding tables, local differential privacy (LDP), and vectorized query execution.

In large-scale biostatistics, epidemiology, and risk modeling, response variables are discrete or non-Gaussian (e.g., binary disease status, Poisson infection counts), modeled via Generalized Linear Models (GLMs):

Fitting GLMs relies on the Iteratively Reweighted Least Squares (IRLS) algorithm, which updates parameters via Fisher scoring:

where $\mathbf{W}_t$ is an $N \times N$ diagonal weight matrix with entries $W_{ii} = \frac{1}{\mathrm{Var}(Y_i) (g'(\mu_i))^2}$, and $\mathbf{z}_t = \mathbf{X} \boldsymbol{\beta}_t + (\mathbf{y} - \boldsymbol{\mu}_t) \odot g'(\boldsymbol{\mu}_t)$ is the working response vector.

Distributed IRLS in Apache Spark MLlib

When dataset $\mathbf{X}$ has $N = 10^8$ rows partitioned across $K$ worker nodes:

1. Each worker node $k$ holds local partition $(\mathbf{X}^{(k)}, \mathbf{y}^{(k)})$.
2. Worker $k$ receives current parameter vector $\boldsymbol{\beta}_t$ from the driver.
3. In a single parallel pass over local data, worker $k$ computes the local $p \times p$ weighted Gram matrix $\mathbf{G}_k$ and local working vector $\mathbf{h}_k$:

1. A Tree-Aggregate network reduction sums the local matrices across all workers:

1. The central driver solves the $p \times p$ linear system $\mathbf{G} \boldsymbol{\beta}_{t+1} = \mathbf{h}$ in $O(p^3)$ time and broadcasts $\boldsymbol{\beta}_{t+1}$ back to the workers.

Because network communication is $O(p^2 \log K)$ per iteration, distributed IRLS fits exact logistic, Poisson, and Gamma GLMs across hundreds of millions of records in 5 to 8 iterations.

To quantify the scalability threshold of distributed statistical computing, we benchmark fitting an $L_2$-regularized logistic regression model on binary classification datasets of escalating row counts ($N \in [10^5, 10^6, 10^7, 10^8]$) with $p = 100$ features, comparing an optimized single-node multicore Python framework (Scikit-Learn with OpenMP vectorization) against a 10-worker Apache Spark cluster: *Key Systems Insight*: Distributed big data frameworks incur non-trivial cluster coordination, serialization, and network startup overheads (roughly 10 to 15 seconds). For datasets that fit comfortably inside local RAM ($N \le 10^6$), optimized single-node multicore tools are 10 to 30 times faster. Distributed Spark clusters should be deployed when data volumes exceed single-machine memory or when processing streaming real-time inputs.

In large-scale distributed computing clusters comprising thousands of commodity servers, hardware failure is not an exceptional anomaly; it is a routine, continuous certainty. If each server has a Mean Time Between Failures (MTBF) of 3 years ($1,095$ days), a cluster containing 10,000 servers experiences an average of nine server crashes every single day.

1. Lineage Graphs vs. Periodic Checkpointing

Classical distributed systems relied on global coordinated checkpointing: pausing computation every 30 minutes to serialize all memory state to distributed disks (HDFS). In statistical learning, checkpointing consumes up to $50\%$ of cluster compute capacity. Apache Spark eliminates checkpointing through Deterministic Lineage Graphs: because transformations (e.g., map, filter) are pure mathematical functions, if worker node 42 fails, Spark re-schedules only the specific lost partition on an idle worker, recomputing it from the original input blocks in seconds.

2. Straggler Mitigation via Speculative Execution

In large distributed matrix operations, algorithmic progress is bounded by the slowest machine in the cluster (The Straggler Problem). Slower execution can be caused by localized network congestion, failing hard drive sectors, or background operating system garbage collection pauses. Modern big data engines deploy Speculative Execution: the master driver monitors task progress across all nodes. If a task runs significantly slower than the median task completion time, the driver launches an identical speculative copy of the task on a different idle node. The node that completes first writes the result, while the redundant straggler task is killed. Speculative execution reduces long-tail latency by $70\%$.

In the physical infrastructure of big data clusters, storage reliability is historically achieved by 3x Replication: every 128 MB block of data in the Hadoop Distributed File System (HDFS) is copied to three physically distinct server racks. While 3x replication guarantees that up to two simultaneous server failures cause zero data loss, it incurs a massive $200\%$ storage overhead (storing 1 Petabyte of statistical data requires 3 Petabytes of physical disk drives).

Modern distributed engines deploy Erasure Coding (EC) using Reed-Solomon codes $\mathrm{RS}(k, m)$ (typically $\mathrm{RS}(6, 3)$):

1. A data file is partitioned into $k = 6$ data blocks.
2. An encoding generator matrix over a Galois field $\mathbb{F}_{2^8}$ computes $m = 3$ parity blocks.
3. The 9 blocks are distributed across 9 independent storage nodes.

Reliability and Cost:

• Any $k = 6$ surviving blocks out of the 9 blocks can reconstruct the original file using Gaussian elimination over finite fields. The system survives any 3 simultaneous disk failures (superior to 3x replication).
• The storage overhead is only $m/k = 3/6 = \mathbf{50\%}$ (storing 1 PB requires only 1.5 PB of physical disk).

Erasure coding slashes data center hardware costs by $50\%$ while guaranteeing multi-petabyte statistical durability.

When engineering distributed big data pipelines for statistical modeling, practitioners must avoid common architectural pitfalls:

1. Avoid Distributed Overkill: Never deploy a distributed Spark or Hadoop cluster for datasets smaller than 10 Gigabytes. Single-node multicore workstations executing vectorized C++/Python routines are significantly faster and incur zero cluster network serialization latency.

2. Prevent Shuffle Spills in Spark DataFrames: Monitor Spark execution logs for shuffle memory spills. If partitions exceed worker RAM, increase the partition count (`spark.sql.shuffle.partitions`) to ensure that every partition fits comfortably in local cache.

3. Leverage Broadcast Joins for Dimension Lookups: Ensure small metadata tables ($< 50 \text{ MB}$) are broadcast to workers via `broadcast(df)` to eliminate expensive network shuffles of multi-terabyte fact tables.