3.2 Interactive Data Preparation with SageMaker Data Wrangler

Q: An ML engineer is preprocessing a dataset containing a high-cardinality categorical feature ('merchant_id' with over 25,000 unique values) to train a gradient boosted tree model. Applying standard One-Hot Encoding in Data Wrangler would result in an excessively wide, sparse matrix that degrades training performance. Which built-in categorical transformation should the engineer apply in Data Wrangler?

Target (Likelihood) Encoding with smoothing. Target Encoding (also called Likelihood Encoding) replaces each categorical level with the expected value of the target variable for that category, with smoothing to prevent overfitting on low-frequency categories. This encodes high-cardinality features into a single informative numeric column without expanding the feature space into tens of thousands of sparse dummy columns. One-Hot Encoding causes the curse of dimensionality, while Robust Scaler and Min-Max Normalization apply only to continuous numeric features, not categorical strings.

Q: A team of data scientists wants to export their finalized feature engineering flow from SageMaker Data Wrangler to a centralized repository where real-time inference endpoints can retrieve the latest features with sub-10ms latency, while batch training jobs can query historical features. Which export target should they choose?

Amazon SageMaker Feature Store. Amazon SageMaker Feature Store is a purpose-built feature repository that provides a dual-store architecture: an Online Store for sub-10ms real-time inference lookups and an S3-backed Offline Store for historical batch training and time-travel querying. Data Wrangler provides native export functionality directly to SageMaker Feature Store feature groups. DynamoDB alone does not provide offline historical Parquet analytics out of the box, Glue DataBrew is an authoring tool rather than a feature store, and Redshift Materialized Views do not provide single-digit millisecond inference APIs.

Key Takeaways

Amazon SageMaker Data Wrangler provides an end-to-end visual workflow for data ingestion, profiling, feature transformation, and pipeline operationalization encapsulated in a .flow file.
Data Wrangler seamlessly connects to multiple disparate sources including Amazon S3, Amazon Athena, Amazon Redshift, AWS Lake Formation, Snowflake, and Databricks.
The Data Quality and Insights Report automatically evaluates dataset health, identifies missing values and outliers, detects target leakage, checks for multicollinearity, and trains a baseline Quick Model.
Over 300 built-in transformations are available (One-Hot, Target Encoding, Robust Scaler, text vectorization), supplemented by custom PySpark, Pandas, and SparkSQL transformation nodes.
Data Wrangler flows can be exported directly into SageMaker Pipelines (ProcessingStep), standalone Python scripts, SageMaker Feature Store feature groups, or AWS Glue ETL jobs.

Last updated: August 2026

Interactive Data Preparation with SageMaker Data Wrangler

Amazon SageMaker Data Wrangler is an integrated visual data preparation tool inside Amazon SageMaker Studio designed to streamline and accelerate the feature engineering lifecycle. Data Wrangler simplifies data exploration, diagnostic quality analysis, transformation authoring, and operationalization into automated MLOps pipelines.

+-----------------------------------------------------------------------------------------+
|                        SAGEMAKER DATA WRANGLER ARCHITECTURE                             |
|                                                                                         |
|   [Data Sources]             [Visual Flow (.flow)]             [Export Targets]         |
|   - Amazon S3                +----------------------+          - SageMaker Pipelines    |
|   - Amazon Athena   ------>  | Import & Sampling    |  ----->  - Python Script / SDK    |
|   - Amazon Redshift          | Quality & Insights   |          - SageMaker Feature Store|
|   - Snowflake / Databricks   | 300+ Transforms / SQL|          - AWS Glue ETL Job       |
|   - Lake Formation           +----------------------+          - Direct S3 Export       |
+-----------------------------------------------------------------------------------------+

1. Core Architecture & Data Ingestion

Data Wrangler organizes all data ingestion steps, transformation sequences, analysis nodes, and export destinations into a Directed Acyclic Graph (DAG) stored as a .flow file (a JSON document detailing the pipeline steps).

Data Ingestion Connectors

Amazon S3: Ingests CSV, Parquet, JSON, and ORC files with automatic schema inference.
Amazon Athena: Query structured data lakes using standard SQL syntax directly from the Data Wrangler console.
Amazon Redshift: Connect to data warehouses with IAM authentication or Secrets Manager credentials.
AWS Lake Formation: Enforces column-level and row-level governance and access control policies during data import.
Third-Party Warehouses: Native connectors for Snowflake and Databricks.

Sampling Strategies During Interactive Exploration

To maintain high UI responsiveness when working with multi-terabyte datasets, Data Wrangler applies intelligent sampling during interactive authoring:

Top K: Reads the first $N$ rows (fastest, ideal for initial schema validation).
Random Sampling: Generates a statistically representative sample across the entire dataset.
Stratified Sampling: Samples proportionally across discrete classes of a specified categorical target column (crucial for imbalanced datasets).

[!NOTE] Interactive Sample vs. Full Job Execution: Interactive transformations in the SageMaker Studio UI execute against the configured sample dataset. However, when you export the flow to a SageMaker Processing Job or Pipeline, the transformation DAG executes across the entire, complete source dataset in distributed fashion.

2. Data Quality and Insights Report

Before applying transformations, ML engineers must evaluate data integrity and detect potential modeling pitfalls. The Data Quality and Insights Report automatically analyzes tabular datasets and produces a comprehensive diagnostic dashboard.

+-----------------------------------------------------------------------------------------+
|                   DATA QUALITY AND INSIGHTS REPORT SECTIONS                             |
|                                                                                         |
|   1. Dataset Summary:       Row/column counts, missing cell %, duplicate rows           |
|   2. Target Column Stats:   Class balance, distribution, entropy, target drift          |
|   3. Quick Model:           Fast baseline model (XGBoost) accuracy & F1 score           |
|   4. Feature Summary:       Min, max, mean, standard deviation, skewness, kurtosis      |
|   5. Target Leakage:        Features that unnaturally predict the target                |
|   6. Multicollinearity:     High cross-feature correlation (Pearson / Spearman)         |
|   7. Anomalous Values:      Statistical outliers detected via isolation forests / IQR   |
+-----------------------------------------------------------------------------------------+

Critical Diagnostic Checks for MLA-C01

1. Target Leakage Detection

Target leakage occurs when a feature contains information that will not be available at real-world inference time, causing artificially inflated training metrics that collapse in production (e.g., a refund_date column used to predict customer churn). Data Wrangler calculates mutual information and predictive correlation between each feature and the target, flagging features that exhibit suspiciously high predictive power.

2. Multicollinearity Analysis

When two or more independent features are highly correlated (e.g., square_feet and number_of_rooms), they introduce redundancy, destabilize linear model coefficients, and inflate variance. Data Wrangler computes a cross-feature correlation matrix and recommends removing or combining redundant features.

3. Quick Model Baseline

Data Wrangler automatically trains an internal baseline model (using gradient boosted decision trees) on the raw features. It provides an immediate benchmark metric (e.g., ROC-AUC, F1-score, or MSE) and surfaces feature importance rankings to guide your feature engineering decisions.

3. Built-In & Custom Transformations

Data Wrangler includes over 300 pre-built transformations categorized by data type and operation:

Common Transformation Categories

Transformation Category	Built-In Operations	Best Practice / ML Impact
Missing Value Handling	Drop missing, Fill with mean/median/mode, Constant replacement, Indicator column	Add a binary indicator column (`col_is_na`) when missingness itself conveys predictive signal (e.g., missing credit score).
Categorical Encoding	One-Hot Encoding, Ordinal/Label Encoding, Target/Likelihood Encoding, Frequency Encoding	Use Target Encoding for high-cardinality features (e.g., ZIP codes) with cross-validation smoothing to prevent overfitting.
Numeric Scaling	Standard Scaler ($Z = \frac{X-\mu}{\sigma}$), Min-Max Scaler ($[0,1]$), Robust Scaler ($IQR$)	Use Robust Scaler when the feature contains significant outliers, as it scales using median and interquartile range.
Text Feature Extraction	Tokenization, Stopword Removal, Vectorization (TF-IDF, Bag of Words, Word Embeddings)	Efficiently converts raw text strings into dense or sparse numerical feature representations.
Datetime Featurization	Extract day of week, hour, cyclical encoding (sine/cosine transforms of cyclical time)	Cyclical sine/cosine transformations preserve the continuous temporal distance between 23:00 and 00:00.

Custom Transformations (Code Nodes)

When built-in transformations are insufficient, Data Wrangler allows adding custom code blocks directly in the DAG:

Custom PySpark: Runs distributed Spark transformations across partitions.
Custom Python (Pandas): Runs standard Pandas transformations on the DataFrame.
Custom SparkSQL: Runs declarative SQL queries (SELECT *, SQRT(colA) AS colA_sqrt FROM df).

# Example Custom PySpark snippet inside a Data Wrangler node
from pyspark.sql.functions import col, sin, cos
import math

# Cyclical encoding for hour of day (0-23)
df = df.withColumn("hour_sin", sin(col("transaction_hour") * (2.0 * math.pi / 24.0)))
df = df.withColumn("hour_cos", cos(col("transaction_hour") * (2.0 * math.pi / 24.0)))

4. Production Export Destinations

Once the transformation flow is finalized in Data Wrangler, it can be exported into various production formats:

+-----------------------------------------------------------------------------------------+
|                        DATA WRANGLER EXPORT DESTINATIONS                                |
|                                                                                         |
|                      [Data Wrangler Flow (.flow)]                                       |
|                                   |                                                     |
|         +-------------------------+-------------------------+                           |
|         |                         |                         |                           |
|         v                         v                         v                           |
|   [SageMaker Pipeline]    [SageMaker Feature Store]  [AWS Glue ETL Job]                 |
|   - Generates             - Ingests features         - Converts .flow                   |
|     ProcessingStep          into Online/Offline        to PySpark Glue                  |
|   - Automated retraining    Feature Groups             ETL script                       |
+-----------------------------------------------------------------------------------------+

Export Options Breakdown

SageMaker Pipelines: Generates a Python script defining a ProcessingStep using the pre-configured SageMaker Data Wrangler container image (sagemaker-data-wrangler-container). The step takes the .flow file and source S3 datasets as inputs and outputs the transformed dataset to S3 for downstream TrainingStep consumption.
SageMaker Feature Store: Automatically creates Feature Group definitions matching the output schema and provisions an ingestion script to populate both the Online and Offline Feature Stores.
Python Code (SageMaker Python SDK): Generates a standalone Jupyter Notebook/script that launches a Processor job to run the flow programmatically.
AWS Glue ETL: Translates the visual DAG steps into an equivalent PySpark script for execution inside an AWS Glue managed ETL job.

# Instantiating a Data Wrangler Processing Step in SageMaker Pipelines
from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker.workflow.steps import ProcessingStep
from sagemaker.processing import Processor

wrangler_processor = Processor(
    role=sagemaker_execution_role,
    image_uri=data_wrangler_image_uri,
    instance_count=2,
    instance_type="ml.m5.4xlarge"
)

step_data_wrangler = ProcessingStep(
    name="CustomerDataWranglerProcessing",
    processor=wrangler_processor,
    inputs=[
        ProcessingInput(
            source="s3://ml-bucket/raw-data/customers.csv",
            destination="/opt/ml/processing/input/customers.csv"
        ),
        ProcessingInput(
            source="s3://ml-bucket/flows/customer_prep.flow",
            destination="/opt/ml/processing/flow"
        )
    ],
    outputs=[
        ProcessingOutput(
            output_name="transformed_features",
            source="/opt/ml/processing/output",
            destination="s3://ml-bucket/features/customer_features/"
        )
    ]
)

5. Cost Optimization & Best Practices

Shut Down Studio Kernel Apps: The interactive Data Wrangler interface runs on dedicated compute instances (e.g., ml.m5.4xlarge). When data preparation authoring is complete, ensure the Data Wrangler application instance is shut down in SageMaker Studio to prevent ongoing hourly compute charges.
Use Sampling for Rapid Iteration: Set appropriate sample sizes (e.g., 50,000 rows) during interactive exploration rather than attempting to render millions of rows in real-time.
Select Appropriate Processing Instance Counts: For export processing jobs, scale instance_count horizontally to parallelize transformations across large S3 multi-part datasets.

Test Your Knowledge

A machine learning engineer uses SageMaker Data Wrangler to inspect a dataset for a binary classification model predicting credit card default. The Data Quality and Insights Report flags a specific feature named 'collection_recovery_fee' with an extremely high predictive score (>0.98) and a target leakage warning. What does this warning indicate, and what action should the engineer take?

The feature contains high multicollinearity with other predictor columns; the engineer should apply standard scaling to normalize its variance.

The feature contains future information that is only populated after a customer defaults; the engineer should drop this feature from the training dataset to prevent train-serve skew.

The feature has high missing value rates; the engineer should use KNN imputation to estimate missing entries.

The feature exhibits severe class imbalance; the engineer should apply SMOTE oversampling.

Test Your Knowledge

A data science team has built a complex data transformation flow in SageMaker Data Wrangler containing custom SQL expressions, missing value imputations, and one-hot encodings. They now need to automate this exact transformation flow to run on weekly incoming raw data batches as part of a recurring, orchestratable SageMaker Pipelines workflow. What is the most efficient and maintainable method to achieve this?

Manually rewrite all Data Wrangler transformation steps in a custom Python Scikit-learn script and run it inside a SageMaker Training Job.

Export the Data Wrangler flow as a SageMaker Pipelines step, which generates a ProcessingStep configured with the Data Wrangler container and the .flow file.

Copy the raw data into Amazon Redshift and execute scheduled stored procedures manually.

Run an interactive SageMaker Studio notebook on a scheduled cron job using a local Pandas script.

Test Your Knowledge

An ML engineer is preprocessing a dataset containing a high-cardinality categorical feature ('merchant_id' with over 25,000 unique values) to train a gradient boosted tree model. Applying standard One-Hot Encoding in Data Wrangler would result in an excessively wide, sparse matrix that degrades training performance. Which built-in categorical transformation should the engineer apply in Data Wrangler?

Target (Likelihood) Encoding with smoothing

One-Hot Encoding with maximum dummy variable expansion

Robust Scaler Normalization

Min-Max Normalization

Test Your Knowledge

A team of data scientists wants to export their finalized feature engineering flow from SageMaker Data Wrangler to a centralized repository where real-time inference endpoints can retrieve the latest features with sub-10ms latency, while batch training jobs can query historical features. Which export target should they choose?

Amazon DynamoDB Global Secondary Index

AWS Glue DataBrew Recipe

Amazon Redshift Materialized View

Amazon SageMaker Feature Store

Up Next

3.3 Feature Management with SageMaker Feature Store

Continue learning

AWS Certified Machine Learning Engineer — Associate

AWS Certified Machine Learning Engineer — Associate (MLA-C01)

3.2 Interactive Data Preparation with SageMaker Data Wrangler

Key Takeaways

Interactive Data Preparation with SageMaker Data Wrangler

1. Core Architecture & Data Ingestion

Data Ingestion Connectors

Sampling Strategies During Interactive Exploration

2. Data Quality and Insights Report

Critical Diagnostic Checks for MLA-C01

1. Target Leakage Detection

2. Multicollinearity Analysis

3. Quick Model Baseline

3. Built-In & Custom Transformations

Common Transformation Categories

Custom Transformations (Code Nodes)

4. Production Export Destinations

Export Options Breakdown

5. Cost Optimization & Best Practices

AWS Certified Machine Learning Engineer — Associate

1Chapter 1: Introduction & Exam Strategy

2Domain 1: Data Ingestion & Storage Architecture for ML

3Domain 1: Feature Engineering & Data Transformation

4Domain 1: Data Quality, Integrity & Ground Truth Labeling

5Domain 2: Model Selection & Framework Architecture

6Domain 2: SageMaker Model Training & Hyperparameter Optimization

7Domain 2: Model Evaluation, Tracking & Model Registry

8Domain 3: SageMaker Inference Options & Endpoint Architecture

9Domain 3: Deployment Strategies & Infrastructure Scaling

10Domain 3: MLOps Pipelines & Workflow Orchestration

11Domain 4: Production Model Monitoring with SageMaker Model Monitor

12Domain 4: Responsible AI, Bias Detection & Governance

13Domain 4: ML Security, Network Isolation & Cost Optimization

AWS Certified Machine Learning Engineer — Associate (MLA-C01)

3.2 Interactive Data Preparation with SageMaker Data Wrangler

Key Takeaways

Interactive Data Preparation with SageMaker Data Wrangler

1. Core Architecture & Data Ingestion

Data Ingestion Connectors

Sampling Strategies During Interactive Exploration

2. Data Quality and Insights Report

Critical Diagnostic Checks for MLA-C01

1. Target Leakage Detection

2. Multicollinearity Analysis

3. Quick Model Baseline

3. Built-In & Custom Transformations

Common Transformation Categories

Custom Transformations (Code Nodes)

4. Production Export Destinations

Export Options Breakdown

5. Cost Optimization & Best Practices