3.3 Comparing Estimators and Transformers

3.3 Comparing Estimators and Transformers

The pyspark.ml library provides a uniform, high-level API built on Spark DataFrames that standardizes machine learning workflows. Inspired by Scikit-learn but engineered for distributed computing, Spark ML organizes end-to-end workflows into modular, reusable Directed Acyclic Graphs (DAGs) known as Pipelines. Mastering the pipeline architecture is essential for building scalable, leakage-free, and production-ready machine learning solutions on Databricks.

---

The Core Abstractions: Transformer, Estimator & Evaluator

Every component in the Spark ML ecosystem derives from three core abstract base classes:

+-----------------------------------------------------------------------------+
|                        SPARK ML CORE ABSTRACTIONS                           |
|                                                                             |
|   +---------------------------------------------------------------------+   |
|   |                             TRANSFORMER                             |   |
|   | Implements: transform(dataset: DataFrame) -> DataFrame              |   |
|   | - Feature Transformer: VectorAssembler, Binarizer                   |   |
|   | - Fitted Model: StandardScalerModel, LogisticRegressionModel        |   |
|   | - Property: Stateless or parameter-frozen; deterministic mapping   |   |
|   +---------------------------------------------------------------------+   |
|                                     ^
|                                     | Produces
|   +---------------------------------+-----------------------------------+   |
|   |                              ESTIMATOR                              |   |
|   | Implements: fit(dataset: DataFrame) -> Transformer                  |   |
|   | - Feature Preprocessor: StringIndexer, StandardScaler, Imputer      |   |
|   | - Learning Algorithm: LogisticRegression, RandomForestClassifier    |   |
|   | - Property: Stateful; learns internal parameters from data          |   |
|   +---------------------------------------------------------------------+   |
|                                                                             |
|   +---------------------------------------------------------------------+   |
|   |                              EVALUATOR                              |   |
|   | Implements: evaluate(dataset: DataFrame) -> float                   |   |
|   | - Metrics Engine: BinaryClassificationEvaluator, RegressionEvaluator|   |
|   | - Property: Computes a scalar performance metric from predictions   |   |
|   +---------------------------------------------------------------------+   |
+-----------------------------------------------------------------------------+

Transformer

A Transformer is an abstraction that transforms one DataFrame into another DataFrame by appending one or more new columns. It implements the method:

$$\text{transform}(\text{dataset}) \to \text{dataset}'$$

• Feature Transformers: Stateless operators such as VectorAssembler (combines individual feature columns into a single vector column) or Binarizer (thresholds continuous values).
• Fitted Model Transformers: Stateful models resulting from an Estimator's fit() call, such as StandardScalerModel (stores mean $\mu$ and standard deviation $\sigma$) or LogisticRegressionModel (stores weight vector $\mathbf{w}$ and intercept $b$). When transform() is called on a trained model, it reads the feature vector and appends rawPrediction, probability, and prediction columns.

Estimator

An Estimator abstracts any algorithm that learns or fits parameters from training data. It implements the method:

$$\text{fit}(\text{dataset}) \to \text{Transformer}$$

• Feature Estimators: StringIndexer inspects the input string column to build an ordered frequency dictionary, returning a StringIndexerModel (Transformer). StandardScaler scans numerical features to compute empirical mean and standard deviation, returning a StandardScalerModel (Transformer).
• Learning Algorithm Estimators: LogisticRegression, RandomForestClassifier, GBTRegressor inspect training feature vectors and target labels to minimize a loss function, returning their respective fitted model Transformer.

Evaluator

An Evaluator computes a quantitative metric evaluating model predictions against true ground truth labels. It implements the method:

$$\text{evaluate}(\text{dataset}) \to \text{scalar metric (float)}$$

• BinaryClassificationEvaluator: Evaluates binary predictions using areaUnderROC (default) or areaUnderPR.
• MulticlassClassificationEvaluator: Evaluates multiclass or binary predictions using f1, accuracy, weightedPrecision, weightedRecall, or logLoss.
• RegressionEvaluator: Evaluates continuous predictions using rmse (default), mse, r2, mae, or mape.

---

The Naming Convention Gives It Away

Spark ML is consistent: an Estimator's fitted output is the same class name with Model appended.

Operators with no Model counterpart are pure Transformers because they require no statistics from the data:

• VectorAssembler — concatenates existing columns into a vector.
• Binarizer — thresholds a numeric column at a value you supply.
• SQLTransformer — applies a SQL statement.
• Tokenizer, StopWordsRemover — deterministic text operations.

OneHotEncoder is the instructive edge case: it is an Estimator in current Spark (producing OneHotEncoderModel) because it must learn the category count in order to size the output vectors.

Why the Distinction Matters in Practice

1. Only Estimators can leak. Because fit() reads data, fitting on the wrong split is what leaks test-set statistics into training. Transformers cannot leak — they apply parameters that already exist.
2. A PipelineModel is entirely Transformers. That is why inference is a single transform() call with no refitting, and why the fitted pipeline can be serialised and shipped to a scoring job.
3. CrossValidator takes an Estimator. It calls fit() on each training fold, so passing a Pipeline (an Estimator) refits the preprocessing per fold — the correct behaviour. Passing an already-fitted PipelineModel would not, and is not accepted.

from pyspark.ml.feature import StringIndexer, VectorAssembler

indexer = StringIndexer(inputCol="contract_type", outputCol="contract_idx")  # Estimator
indexer_model = indexer.fit(train_df)                                        # -> Transformer
indexed_df = indexer_model.transform(train_df)                               # DataFrame -> DataFrame

assembler = VectorAssembler(inputCols=["contract_idx", "tenure"], outputCol="features")
assembled_df = assembler.transform(indexed_df)   # Transformer: no fit() needed or available

Calling .transform() on an unfitted Estimator raises an error, and calling .fit() on a pure Transformer is not part of its API — a distinction exam code snippets test directly.

The pattern that generalises

Any Spark ML class that must learn something from the data before it can act is an Estimator, and what it produces is a Transformer whose name is normally the Estimator's name plus Model: StringIndexer becomes StringIndexerModel, StandardScaler becomes StandardScalerModel, LogisticRegression becomes LogisticRegressionModel, Imputer becomes ImputerModel. Classes that need nothing from the data — such as VectorAssembler, Tokenizer, SQLTransformer, and Binarizer with a fixed threshold — are Transformers from the moment they are constructed. Asking "does this step have to look at the training data first?" answers the classification question for any class the exam names, including ones you have never used: an imputer must learn the column means, so it is an Estimator; an assembler merely concatenates existing columns into a vector, so it is a Transformer.