Apache Spark

Spark SQL parses SQL statements into an Abstract Syntax Tree (AST), passes them through the Analyzer for logical plan resolution (type checking, catalog resolution, predicate pushdown), then applies Catalyst optimizer rules to transform logical plans into optimized physical execution plans. The optimizer uses cost-based and rule-based strategies to select optimal join orders, partition pruning, and columnar execution paths. Physical plans are executed via SparkPlan's distributed task scheduling across cluster nodes.

Spark Core provides RDD (Resilient Distributed Dataset) and DataFrame abstractions that partition data across cluster nodes and apply transformations (map, filter, join, groupBy) lazily. Transformations build a Directed Acyclic Graph (DAG) of operations; only when an action (collect, write, count) is called does the DAG Scheduler convert the DAG into stages, optimize shuffle boundaries, and dispatch tasks to executors. Lineage tracking enables fault tolerance via RDD recomputation on node failure.

Spark's Declarative Streaming Pipelines (SDP) enable users to define streaming dataflow graphs declaratively, specifying sources, transformations, and sinks as a DAG. The SDP compiler converts the dataflow graph into a Spark Structured Streaming job, optimizing the graph for execution. This abstraction sits above Structured Streaming, providing a higher-level API for common streaming patterns (windowing, stateful aggregations, joins). The SDP Python API and CLI enable non-Scala users to define pipelines without writing Scala code.

Spark's Variant type enables efficient storage and querying of semi-structured data (JSON, nested objects) without requiring a fixed schema. Variant columns store data in a compact binary format that preserves type information and enables efficient path-based access (e.g., variant_col['key']['nested_key']). The Variant type supports schema evolution; new fields can be added without rewriting existing data. Queries on Variant columns are optimized via Catalyst; filters and projections are pushed down to the Variant reader, avoiding full deserialization.

Spark SQL integrates with Hive metastore (or Spark's built-in catalog) to store table metadata (schema, location, partitions, statistics). The Thrift server enables JDBC/ODBC clients (e.g., Tableau, SQL clients) to connect to Spark as if it were a Hive server, executing SQL queries via the same Catalyst optimizer. Partition pruning uses metastore statistics to skip partitions; table statistics enable cost-based join optimization. Spark can read/write Hive tables directly, enabling migration from Hive to Spark without data movement.

Pandas API on Spark (pyspark.pandas) provides a Pandas-compatible API that maps Pandas operations to Spark DataFrames, enabling data scientists familiar with Pandas to scale their code to distributed datasets without learning Spark API. Operations like groupby, merge, apply are translated to Spark SQL/DataFrame operations and executed distributedly. The API handles schema inference, type conversion, and result collection transparently. This enables code portability: Pandas code can be scaled to Spark by changing import statements.

Spark Structured Streaming treats streaming data as an unbounded table, applying the same SQL/DataFrame operations as batch processing. Micro-batches are processed at fixed intervals; the Catalyst optimizer generates physical plans for each batch. Stateful operations (aggregations, joins with state) use the StateStore interface backed by RocksDB for fault-tolerant state persistence. Checkpointing writes offset metadata and state snapshots to distributed storage; on failure, the system replays from the last checkpoint to recover state exactly-once semantics.

PySpark provides a Python-native DataFrame API that mirrors Scala/SQL semantics but executes in the JVM via Py4J (inter-process communication). Recent versions support Spark Connect, a gRPC-based client-server architecture where Python code runs in a separate process and communicates with a Spark server, eliminating JVM overhead in the Python process. Arrow serialization (PyArrow) enables efficient columnar data transfer between Python and JVM, reducing serialization overhead by 10-100x vs pickle. User-Defined Functions (UDFs) can be vectorized (Pandas UDFs) to process batches of rows in Python, amortizing JVM/Python boundary crossing costs.

Spark MLlib provides distributed implementations of classic ML algorithms (linear regression, logistic regression, decision trees, random forests, k-means, ALS) that partition training data across cluster nodes and use iterative optimization (SGD, L-BFGS) to converge on model parameters. The ML Pipeline API (higher-level than RDD-based MLlib) chains transformers (feature scaling, encoding) and estimators (model training) into a DAG, enabling reproducible feature engineering and model training. Pipelines serialize to disk for production serving. Feature transformers (StandardScaler, OneHotEncoder, VectorAssembler) operate on DataFrames, integrating with Spark SQL.

GraphX represents graphs as RDDs of vertices and edges, enabling distributed graph algorithms via the Pregel abstraction (vertex-centric programming model). Algorithms like PageRank, connected components, and triangle counting are implemented as iterative message-passing between vertices; each iteration sends messages to neighboring vertices, aggregates incoming messages, and updates vertex state. The VertexRDD and EdgeRDD abstractions optimize storage and communication by partitioning vertices/edges across cluster nodes. Graph operations (subgraph, mapVertices, mapEdges) are lazy and optimized via Spark's DAG scheduler.

Adaptive Query Execution monitors query execution at runtime, collecting statistics (partition sizes, data skew) after each stage completes, then re-optimizes subsequent stages based on actual data distribution. AQE dynamically adjusts join strategies (broadcast join vs shuffle join) if partition sizes change, coalesces small partitions to reduce task overhead, and skew-aware joins detect and handle data skew by splitting large partitions. The optimizer re-plans the remaining query DAG after each stage, enabling decisions based on real data rather than pre-execution estimates.

Spark SQL executes queries in columnar format (not row-by-row), storing data as arrays of values per column. Parquet files are read via vectorized readers that load entire column chunks into memory and process them as vectors, enabling CPU cache efficiency and SIMD (Single Instruction Multiple Data) operations. The Columnar Batch abstraction holds multiple rows of columnar data; operators (filter, projection, aggregation) process batches instead of individual rows, reducing function call overhead. Columnar execution is transparent to users but dramatically improves performance for analytical queries (10-100x faster than row-based execution for selective filters).

Spark Connect decouples the Spark client (Python, Scala, R) from the Spark server via gRPC, enabling lightweight client processes to submit queries and receive results without embedding a JVM. The client serializes DataFrame operations into a logical plan protobuf message, sends it to the server, and the server executes the plan using Catalyst optimizer and physical execution engine. Results are streamed back to the client via Arrow format. This architecture enables Spark to run in serverless environments (AWS Lambda, Google Cloud Functions) where JVM overhead is prohibitive, and supports multiple clients connecting to a single Spark server.

Spark's shuffle operation (required for joins, groupBy, repartition) partitions data across nodes and sorts within each partition. When data exceeds executor memory, Spark spills intermediate results to disk using an external sort algorithm (similar to merge sort), reading back sorted chunks and merging them. The ExternalSorter class manages this process transparently; developers don't need to tune spill thresholds. Shuffle writes are compressed (LZ4, Snappy) and checksummed; the shuffle service on each node serves blocks to downstream tasks, enabling efficient data transfer.