4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus

Q: A financial institution uses a private workforce of 5 annotators to label complex fraud transaction narratives. Because the narratives are nuanced, worker opinions frequently diverge. The ML engineer wants to use an annotation consolidation method that models individual worker accuracy and error matrices rather than treating all worker votes equally. Which annotation consolidation method does SageMaker Ground Truth provide for this requirement?

The Dawid-Skene algorithm using Expectation-Maximization. The Dawid-Skene algorithm is an Expectation-Maximization (EM) probabilistic model used by SageMaker Ground Truth for annotation consolidation. It simultaneously estimates the true class labels and the individual confusion/error matrices for each worker, giving higher weight to workers who demonstrate consistent historical accuracy across tasks rather than relying on naive majority voting where all workers are treated as equally reliable.

Key Takeaways

Amazon SageMaker Ground Truth supports 4 distinct workforce types: Public (Amazon Mechanical Turk for non-sensitive data), Private (internal staff authenticated via Amazon Cognito or OIDC), Vendor (vetted AWS Marketplace vendors), and Ground Truth Plus (turnkey managed labeling service).
Built-in labeling task modalities include Image Classification, Object Detection (Bounding Boxes), Semantic Segmentation, Text Classification, Named Entity Recognition (NER), and 3D Point Cloud Labeling, with custom workflows supported via Liquid HTML and AWS Lambda.
Ground Truth Active Learning (Automated Data Labeling) significantly reduces labeling costs by training an auto-labeling model that automatically annotates high-confidence samples while routing ambiguous, low-confidence samples to human annotators.
Annotation consolidation resolves multi-worker disagreements using probabilistic algorithms such as Dawid-Skene and expectation-maximization (EM) alongside golden calibration datasets.
Labeled outputs are generated as Augmented Manifest files (.jsonl), which can be fed directly into SageMaker Training Channels using the AugmentedManifestFile S3 data type without intermediate data conversion.

Last updated: August 2026

Data Labeling with SageMaker Ground Truth & Ground Truth Plus

Supervised machine learning algorithms require large volumes of accurately annotated training data. Generating high-quality ground truth labels is often the most time-consuming and expensive bottleneck in the ML lifecycle. Amazon SageMaker Ground Truth and Amazon SageMaker Ground Truth Plus provide managed human-in-the-loop data labeling services that streamline dataset creation across computer vision, natural language processing, audio, and 3D point cloud domains.

For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master the differences between the four workforce types, active learning and automated labeling mechanics, annotation consolidation algorithms (such as Dawid-Skene), custom labeling workflow architectures using AWS Lambda, and ingesting Augmented Manifest files directly into SageMaker training jobs.

+---------------------------------------------------------------------------------------------------+
|                         SAGEMAKER GROUND TRUTH WORKFLOW SPECTRUM                                  |
|                                                                                                   |
|   [Raw Unlabeled Data] ---> [Ground Truth Labeling Job]                                           |
|                                    |                                                              |
|             +----------------------+----------------------+                                       |
|             | Active Learning (Auto-Labeling)             | Workforce Selection                   |
|             v                                             v                                       |
|   [Auto-Labeling Model]                         [1. Public (Mechanical Turk)]                     |
|   - Confident >= Threshold ---> Auto-labeled    [2. Private (Cognito / OIDC)]                     |
|   - Confident <  Threshold ---> Routed to Human [3. Vendor (AWS Marketplace)]                     |
|                                                 [4. Ground Truth Plus (Managed)]                  |
|                                                           |                                       |
|                                                           v                                       |
|                                             [Annotation Consolidation]                            |
|                                             (Dawid-Skene / Consensus)                             |
|                                                           |                                       |
|                                                           v                                       |
|                                             [Augmented Manifest (.jsonl)]                         |
|                                                           |                                       |
|                                                           v                                       |
|                                             [SageMaker Training Job]                              |
+---------------------------------------------------------------------------------------------------+

1. Built-in Task Types & Custom Workflows

Amazon SageMaker Ground Truth provides out-of-the-box labeling user interfaces (worker UIs) for the most common ML tasks:

Built-in Modalities & Task Types

Modality	Task Type	Worker Interface & Output Description
Computer Vision	Image Classification (Single/Multi-label)	Assigns one or more categorical tags to whole images
Computer Vision	Object Detection (Bounding Box)	Draws rectangular 2D bounding boxes around specified classes with coordinates `[top, left, height, width]`
Computer Vision	Semantic Segmentation	Pixel-level mask segmentation to classify every pixel in an image (e.g., road, vehicle, pedestrian)
Natural Language	Text Classification	Categorizes paragraphs or documents (e.g., sentiment, topic classification)
Natural Language	Named Entity Recognition (NER)	Highlights and tags specific token spans in text (e.g., PERSON, ORGANIZATION, DATE, MEDICAL_CODE)
3D Point Cloud	3D Bounding Box & Tracking	Labels 3D cuboids and tracks dynamic objects across LiDAR point cloud frames
Video	Video Object Tracking / Detection	Bounding boxes tracked across sequential video frames with automatic interpolation

Custom Labeling Workflows

When requirements exceed built-in task types (e.g., combined text-image QA, audio transcript annotation, or complex multi-step forms), ML engineers create Custom Workflows using two integration hooks:

Pre-Annotation Lambda: Preprocesses raw input data from S3 before it is displayed to the worker (e.g., tokenizing text, attaching reference metadata, or dynamically fetching external assets).
Custom Worker UI (Liquid HTML): Built using HTML, CSS, JavaScript, and Crowd HTML Elements with Liquid templating tags (e.g., <crowd-bounding-box src='{{ task.input.source-ref | grant_read_access }}' ...>).
Post-Annotation Lambda (Consolidation): Executes custom consolidation algorithms over multiple worker responses to produce a single ground truth record.

+---------------------------------------------------------------------------------------------------+
|                             CUSTOM WORKFLOW ARCHITECTURE                                          |
|                                                                                                   |
|   [Raw S3 Object] ---> [Pre-Annotation Lambda] ---> [Custom Worker HTML UI (Liquid)]              |
|                                                            |                                      |
|                                                            v                                      |
|   [Final Manifest] <--- [Post-Annotation Lambda] <--- [Worker Annotations (1..N)]                 |
+---------------------------------------------------------------------------------------------------+

2. Workforce Management & Security Governance

Choosing the correct workforce is a critical architectural and regulatory decision on the MLA-C01 exam.

+---------------------------------------------------------------------------------------------------+
|                                 WORKFORCE SELECTION MATRIX                                        |
|                                                                                                   |
|   Workforce Type          Identity Provider          Best For                     PII / Sensitive |
|   ---------------------   ------------------------   --------------------------   --------------- |
|   Amazon Mechanical Turk  Public AWS Crowdsource     Large non-sensitive datasets STRICTLY NO     |
|   Private Workforce       Amazon Cognito / OIDC      Internal staff, confidential ALLOWED (Secure)|
|   Vendor Workforce        AWS Marketplace Vendors    Specialized domain expertise ALLOWED (SLA)   |
|   Ground Truth Plus       Managed Turnkey AWS Team   Zero-ops custom workflows    ALLOWED (HIPAA) |
+---------------------------------------------------------------------------------------------------+

Detailed Workforce Characteristics

1. Amazon Mechanical Turk (Public Workforce)

A global on-demand crowdsource community of over 500,000 workers.
Advantages: Highly scalable, fastest turnaround, lowest cost per task.
Strict Constraint: NEVER use Mechanical Turk for datasets containing Personally Identifiable Information (PII), Protected Health Information (PHI), financial data, or proprietary intellectual property. Workers are anonymous public individuals.

2. Private Workforce

Composed of internal company employees, subject matter experts, or contracted contractors.
Authentication & Federation: Managed via Amazon Cognito User Pools (standalone worker portal) or federated with corporate Identity Providers (e.g., Okta, Microsoft Entra ID / Azure AD, PingFederate) using OpenID Connect (OIDC) or SAML 2.0.
Security: Fully compliant with enterprise data protection policies. Restrictable to corporate VPC networks using AWS PrivateLink.

3. Vendor Workforce

Professional labeling service providers vetted by AWS and contracted directly through the AWS Marketplace.
Advantages: Access to specialized domain experts (e.g., radiologists for medical imaging or attorneys for legal contract analysis) backed by Service Level Agreements (SLAs) for labeling accuracy and turnaround time.

4. Amazon SageMaker Ground Truth Plus

A fully managed, turnkey data labeling service where AWS manages the entire labeling operation.
AWS ML experts build custom labeling workflows, create worker training materials, assemble and manage pre-vetted domain-expert workforces, and enforce multi-stage quality assurance.
Key Benefit: Requires zero data labeling infrastructure or UI template development from your internal engineering team.

3. Automated Data Labeling & Active Learning

Automated Data Labeling in SageMaker Ground Truth utilizes an Active Learning feedback loop to minimize the number of human annotations required, reducing labeling costs by up to 70%.

+---------------------------------------------------------------------------------------------------+
|                         ACTIVE LEARNING AUTOMATED LABELING LOOP                                   |
|                                                                                                   |
|   [1. Raw S3 Objects]                                                                             |
|           |                                                                                       |
|           v                                                                                       |
|   [2. Human Labeling Phase] ---> Human workers label initial random batch (e.g., 20%)             |
|           |                                                                                       |
|           v                                                                                       |
|   [3. Auto-Labeling Model]  ---> SageMaker trains an internal model on human-labeled data         |
|           |                                                                                       |
|           v                                                                                       |
|   [4. Batch Inference]      ---> Model predicts labels on remaining unlabeled data                |
|           |                                                                                       |
|           +-----------------------------------+                                                   |
|           | Confidence >= Threshold           | Confidence < Threshold                            |
|           v                                   v                                                   |
|   [Auto-Labeled to Manifest]        [5. Routed to Human Workers]                                  |
|                                               |                                                   |
|                                               v                                                   |
|                                     [6. Retrain Auto-Model] ---> (Iterative Loop Repeats)         |
+---------------------------------------------------------------------------------------------------+

Operational Mechanics of Active Learning

Bootstrap Phase: A random sample of raw data is sent to human annotators.
Model Training: Once a sufficient volume of human annotations is accumulated (minimum requirements: at least 1,250 validation objects for image classification, 5,000 for text), Ground Truth trains a custom neural network auto-labeling model.
Inference & Scoring: The auto-labeling model scores the remaining unlabeled objects.
Confidence Partitioning:
- Objects whose model prediction confidence exceeds the pre-calibrated threshold are automatically assigned labels and written directly to the output manifest (human-annotated: 'no').
- Objects below the confidence threshold are dispatched to human annotators (human-annotated: 'yes').
Iterative Retraining: Newly human-labeled samples are merged back into the training pool to retrain the auto-labeling model across subsequent iterations, progressively raising model confidence and decreasing human workload.

4. Quality Control & Annotation Consolidation

To ensure ground truth integrity, multiple workers typically label the same object (e.g., 3 to 5 workers per image). Ground Truth uses mathematical consolidation algorithms to resolve conflicting worker inputs.

+---------------------------------------------------------------------------------------------------+
|                                 ANNOTATION CONSOLIDATION METHODS                                  |
|                                                                                                   |
|   1. Majority Voting:                                                                             |
|      - Takes the mode of worker labels. Naive approach; treats all workers as equally reliable.   |
|                                                                                                   |
|   2. Dawid-Skene Algorithm (Expectation-Maximization):                                            |
|      - Jointly estimates worker confusion matrices (individual worker accuracy) and true class   |
|        probabilities simultaneously.                                                              |
|      - Gives higher probabilistic weight to consistently accurate annotators.                     |
|                                                                                                   |
|   3. Golden Datasets / Worker Calibration:                                                        |
|      - Injects known ground truth verification tasks into worker queues.                          |
|      - Measures worker accuracy in real time; disqualifies workers who fail calibration checks.   |
+---------------------------------------------------------------------------------------------------+

The Dawid-Skene Algorithm

Rather than assuming every worker is equally skilled, the Dawid-Skene algorithm uses an iterative Expectation-Maximization (EM) framework:

E-step: Estimates the true class probability distribution for each object based on current worker accuracy estimates.
M-step: Updates the individual error matrix (confusion matrix) for each worker based on the estimated true labels.
This ensures that if Worker A has a 99% historical accuracy while Worker B has a 60% accuracy, Worker A's label carries significantly more mathematical weight during consolidation.

5. Output Manifests & SageMaker Training Integration

Ground Truth outputs annotations in the form of an Augmented Manifest file (output.manifest), formatted as a newline-delimited JSON (.jsonl) file stored in Amazon S3.

Structure of an Augmented Manifest File

Each line represents a single data record containing the S3 source reference, the consolidated label, and detailed metadata:

{
  'source-ref': 's3://ml-training-lake/images/sample_001.jpg',
  'bounding-box-labels': {
    'image_size': [{'width': 1920, 'height': 1080, 'depth': 3}],
    'annotations': [
      {'class_id': 0, 'left': 340, 'top': 180, 'width': 220, 'height': 450}
    ]
  },
  'bounding-box-labels-metadata': {
    'class-map': {'0': 'pedestrian'},
    'human-annotated': 'yes',
    'confidence': 0.96,
    'creation-date': '2026-08-16T12:00:00.000000',
    'type': 'groundtruth/object-detection',
    'job-name': 'gt-pedestrian-detection-job'
  }
}

Ingesting Augmented Manifests into SageMaker Training Jobs

SageMaker built-in algorithms (such as Object Detection, Image Classification, and Semantic Segmentation) and custom Script Mode estimators can consume Augmented Manifest files directly using the TrainingInput configuration with s3_data_type='AugmentedManifestFile':

import sagemaker
from sagemaker.inputs import TrainingInput

# Configure TrainingInput channel pointing to Ground Truth Augmented Manifest
train_channel = TrainingInput(
    s3_data='s3://ml-training-lake/ground-truth-output/gt-job/manifests/output/output.manifest',
    content_type='application/x-recordio',
    s3_data_type='AugmentedManifestFile',
    record_wrapping='RecordIO',
    attribute_names=['source-ref', 'bounding-box-labels']  # Identifies input and label keys
)

# Initialize Estimator
estimator = sagemaker.estimator.Estimator(
    image_uri=sagemaker.image_uris.retrieve('object-detection', 'us-east-1'),
    role=sagemaker.get_execution_role(),
    instance_count=1,
    instance_type='ml.g5.2xlarge',
    output_path='s3://ml-training-lake/models/'
)

# Fit estimator using the manifest channel
estimator.fit({'train': train_channel})

[!IMPORTANT] Attribute Names Alignment: When passing an Augmented Manifest to a SageMaker training channel, the attribute_names parameter must list exactly two JSON keys: the source reference key (e.g., source-ref) and the label key (e.g., bounding-box-labels). SageMaker streams the raw objects and annotations directly without requiring manual data conversion.

[!NOTE] Amazon Augmented AI (Amazon A2I): Where Ground Truth labels raw training data, Amazon A2I inserts human review into production predictions — routing low-confidence or high-risk inferences (for example, Amazon Textract document extractions, Amazon Rekognition moderation results, or a custom model score below a confidence threshold) to human reviewers before the output is consumed downstream. Choose A2I when an exam scenario requires a human-in-the-loop review step on live model outputs rather than dataset annotation.

Loading diagram...

SageMaker Ground Truth End-to-End Architecture

Test Your Knowledge

A hospital network is preparing a dataset of 50,000 chest X-ray scans to train a deep learning diagnostic model. The scans contain Protected Health Information (PHI) and patient metadata that must strictly comply with HIPAA regulations. The hospital lacks the internal staff to label 50,000 images and wants to engage certified medical radiologists with minimal administrative overhead. Which workforce strategy should the ML engineer select?

Amazon Mechanical Turk with automated worker qualification filters.

Amazon SageMaker Ground Truth with a public crowdsource workforce and active learning enabled.

Amazon SageMaker Ground Truth with a Vendor Workforce contracted via AWS Marketplace or Amazon SageMaker Ground Truth Plus.

Amazon Mechanical Turk with an AWS Lambda pre-annotation function to anonymize images.

Test Your Knowledge

An autonomous driving startup is labeling 500,000 camera frames for 2D vehicle and pedestrian bounding box detection. To minimize labeling costs, the ML engineer enables Automated Data Labeling in SageMaker Ground Truth. How does the Active Learning mechanism operate during the labeling job?

Ground Truth trains an internal ML model on an initial human-annotated batch, runs inference on the remaining data, automatically labels high-confidence samples, routes low-confidence samples to human annotators, and iteratively retrains the model.

Ground Truth applies pre-trained ImageNet weights to automatically label 100% of the images and only sends images to humans if the image resolution is below 1080p.

Ground Truth uses unsupervised k-means clustering to group similar images and assigns human labels from cluster centroids to all cluster members.

Ground Truth sends all 500,000 images to human annotators simultaneously and uses an auto-labeling model to calculate worker payment amounts.

Test Your Knowledge

An ML team has completed a Ground Truth object detection labeling job. The resulting annotations are saved in Amazon S3 as an Augmented Manifest file (output.manifest). The team now wants to train a SageMaker built-in Object Detection algorithm using this file directly without converting it to individual RecordIO binary files. How should the ML engineer configure the SageMaker Estimator training channel in the Python SDK?

Set s3_data_type='S3Prefix' and provide the S3 directory containing the raw unannotated images.

Convert the JSON manifest to a CSV file using AWS Glue and pass content_type='text/csv'.

Set s3_data_type='ManifestFile' and pass the S3 URI of a custom JSON manifest containing image bounding box matrices.

Configure s3_data_type='AugmentedManifestFile' in TrainingInput, supply the S3 URI of output.manifest, and specify attribute_names=['source-ref', 'bounding-box-labels'].

Test Your Knowledge

A financial institution uses a private workforce of 5 annotators to label complex fraud transaction narratives. Because the narratives are nuanced, worker opinions frequently diverge. The ML engineer wants to use an annotation consolidation method that models individual worker accuracy and error matrices rather than treating all worker votes equally. Which annotation consolidation method does SageMaker Ground Truth provide for this requirement?

Simple Majority Rule Voting

The Dawid-Skene algorithm using Expectation-Maximization

Random Selection Consensus

K-Nearest Neighbors Worker Averaging

Up Next

5.1 SageMaker Built-in Algorithms & Selection Criteria

Domain 2: Model Selection & Framework Architecture

AWS Certified Machine Learning Engineer — Associate

1Chapter 1: Introduction & Exam Strategy

2Domain 1: Data Ingestion & Storage Architecture for ML

3Domain 1: Feature Engineering & Data Transformation

4Domain 1: Data Quality, Integrity & Ground Truth Labeling

5Domain 2: Model Selection & Framework Architecture

6Domain 2: SageMaker Model Training & Hyperparameter Optimization

7Domain 2: Model Evaluation, Tracking & Model Registry

8Domain 3: SageMaker Inference Options & Endpoint Architecture

9Domain 3: Deployment Strategies & Infrastructure Scaling

10Domain 3: MLOps Pipelines & Workflow Orchestration

11Domain 4: Production Model Monitoring with SageMaker Model Monitor

12Domain 4: Responsible AI, Bias Detection & Governance

13Domain 4: ML Security, Network Isolation & Cost Optimization

AWS Certified Machine Learning Engineer — Associate (MLA-C01)

4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus

Key Takeaways

Data Labeling with SageMaker Ground Truth & Ground Truth Plus

1. Built-in Task Types & Custom Workflows

Built-in Modalities & Task Types

Custom Labeling Workflows

2. Workforce Management & Security Governance

Detailed Workforce Characteristics

1. Amazon Mechanical Turk (Public Workforce)

2. Private Workforce

3. Vendor Workforce

4. Amazon SageMaker Ground Truth Plus

3. Automated Data Labeling & Active Learning

Operational Mechanics of Active Learning

4. Quality Control & Annotation Consolidation

The Dawid-Skene Algorithm

5. Output Manifests & SageMaker Training Integration

Structure of an Augmented Manifest File

Ingesting Augmented Manifests into SageMaker Training Jobs