Apache Spark is the most widely used distributed data processing engine in the industry. Its two fundamental abstractions, DataFrames and pipelines, enable transforming, analysing, and modelling large data volumes with expressive code, automatically optimised by the Catalyst engine and executed in parallel across node clusters. The project has been under active development for over a decade: it became an Apache Software Foundation top-level project in 2014, according to Wikipedia[1].

Key takeaways

  • Spark DataFrames are distributed, schematised tables that can be queried with SQL or the Python/Scala functional API.

  • Transformations in Spark are lazy: they are not executed until an action is needed, allowing the engine to optimise the execution plan.

  • Pipelines organise transformations and models into a reproducible chain, essential for ML in production.

  • Partitioning and caching are the two most direct performance optimisation levers.

  • Spark scales horizontally: adding nodes to the cluster increases capacity without rewriting code.

DataFrames: Spark’s central abstraction

A Spark DataFrame is a distributed collection of data organised into named, typed columns, conceptually similar to a relational database table or a pandas DataFrame, but distributed across cluster nodes.

Spark DataFrames can be created from multiple sources:

  • CSV, JSON, Parquet, and ORC files.

  • Relational databases via JDBC.

  • Storage systems like S3, HDFS, or Azure Data Lake.

  • Real-time streams (Spark Structured Streaming).

The most common operations are:

  • Filtering (filter, where): selecting rows that meet a condition.

  • Transformation (select, withColumn): adding or modifying columns with expressions.

  • Aggregation (groupBy, agg): computing metrics per group.

  • Join: combining two DataFrames by one or more common keys.

A fundamental property is immutability: each transformation creates a new DataFrame; it does not modify the existing one. This makes pipelines reproducible and facilitates reasoning about data flow.

Lazy execution and the Catalyst optimiser

Spark does not execute transformations immediately. When you chain .filter(), .groupBy(), and .select(), Spark builds a logical plan of the query. Only when you call an action (.show(), .collect(), .write()) does Spark deliver that plan to the Catalyst optimiser, described in Databricks’ Catalyst Optimizer glossary entry[2], which:

  • Analyses the logical plan and checks schemas.

  • Generates multiple alternative physical plans.

  • Selects the one with the lowest estimated cost (based on data statistics).

  • Generates JVM code (or columnar code with Tungsten) to execute it.

This deferred execution is what allows Spark to optimise sequences of transformations that, if executed step by step, would be inefficient. It is analogous to a database query optimiser, but for distributed code. Since version 3.0, released in 2020, Spark adds Adaptive Query Execution (AQE), which re-tunes the physical plan at runtime using real per-stage statistics instead of relying only on upfront estimates.

Pipelines: reproducibility in production

In the context of machine learning with Spark MLlib, a pipeline is an ordered sequence of stages (Stages) where each takes an input DataFrame, transforms it, and produces an output DataFrame:

from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.classification import RandomForestClassifier

assembler = VectorAssembler(inputCols=["feature1", "feature2"], outputCol="features")
scaler = StandardScaler(inputCol="features", outputCol="scaled_features")
rf = RandomForestClassifier(featuresCol="scaled_features", labelCol="label")

pipeline = Pipeline(stages=[assembler, scaler, rf])
model = pipeline.fit(train_df)
predictions = model.transform(test_df)

The advantages of pipelines are:

  • Reproducibility: the same Pipeline object applied to new data produces exactly the same transformation flow.

  • Serialisation: trained models can be saved to disk and loaded without retraining.

  • Cross-validation integration: CrossValidator and TrainValidationSplit in Spark MLlib work natively with pipelines for hyperparameter selection.

Optimisation: partitioning and caching

Two techniques have the greatest impact on real performance:

Partitioning. Spark divides data into partitions processed in parallel. Inadequate partitioning creates bottlenecks:

  • Too few partitions: underutilises the cluster; nodes are idle.

  • Too many small partitions: coordination overhead outweighs the parallelism benefit.

  • Skew: if one partition concentrates 80% of the data (e.g., a frequent value in the join key), that node becomes the bottleneck.

repartition(n) redistributes data with a full shuffle; coalesce(n) reduces partition count without a shuffle, useful just before writing the result. The official Spark performance tuning guide[3] sets the default value of spark.sql.shuffle.partitions at 200, a figure meant for generic clusters that almost always needs adjusting to the real data size.

Data caching. When a DataFrame is used multiple times in the same flow (e.g., as the basis for more than one aggregation), persist it in memory with .cache() or .persist(). Without caching, Spark recomputes the DataFrame from source every time it is referenced. With caching, the read cost is paid only once.

For use cases where Spark connects with broader machine learning systems, the LazyPredict pattern for rapid model comparison is a natural complement once the dataset is prepared. It also parallels big data use in decision-making.

Conclusion

Spark DataFrames and pipelines are the standard infrastructure for processing data at scale in production environments. The key to performance is not the tool itself: it is correctly designing the partitioning, using caching strategically, and letting the Catalyst optimiser do its work. A well-built Spark pipeline is code that scales from a laptop to a hundreds-of-nodes cluster without structural changes.

You can read this article in Spanish: Dataframes y Pipelines en Spark: Optimización de Procesamiento de Datos.

Sources:

  1. Wikipedia: Apache Spark[1]
  2. Databricks: Catalyst Optimizer[2]
  3. Apache Spark: SQL Performance Tuning Guide[3]

Frequently asked questions

Why does my Spark job do nothing until I call `.show()` or `.write()`?

Because transformations are lazy: when you chain .filter(), .groupBy() and .select(), Spark only builds a logical plan of the query. Only when you invoke an action such as .show(), .collect() or .write() does it hand that plan to the Catalyst optimiser. Catalyst checks schemas, generates alternative physical plans, picks the one with the lowest estimated cost and produces the JVM code (or columnar code with Tungsten) that runs. Since Spark 3.0, Adaptive Query Execution also re-tunes the physical plan at runtime using real per-stage statistics.

When should I use `repartition` versus `coalesce` in Spark?

repartition(n) redistributes data with a full shuffle, useful for increasing parallelism or fixing skew when one partition concentrates, say, 80% of the data because of a frequent join-key value. coalesce(n) reduces the partition count without a shuffle and is best used just before writing the result. Too few partitions underutilise the cluster and too many small ones make coordination overhead outweigh the parallelism; the default spark.sql.shuffle.partitions is 200 and almost always needs adjusting to the real data size.

When is it worth calling `.cache()` on a DataFrame?

When the same DataFrame is used more than once in the same flow, for instance as the basis for more than one aggregation. Without caching, Spark recomputes the DataFrame from source every time it is referenced, because each transformation creates a new immutable DataFrame; with .cache() or .persist() the read cost is paid only once. Persist the cleaned DataFrame once and then derive a groupBy and an additional filter from it without re-reading from disk.

Sources

  1. Wikipedia
  2. Databricks’ Catalyst Optimizer glossary entry
  3. official Spark performance tuning guide