4.1 Pipeline Fundamentals & Core Stages ($match, $project, $addFields)

4.1 Pipeline Fundamentals & Core Stages ($match, $project, $addFields)

The MongoDB Aggregation Framework is a declarative, multi-stage data processing pipeline designed for advanced analytics, real-time data transformations, and reporting. While standard CRUD query methods (find() and findOne()) excel at simple document retrieval, filtering, and basic sorting, the Aggregation Framework allows developers to express complex data transformations, multi-document computations, and reshaping operations directly on the database server.

Understanding the mechanics of pipeline stage execution, index utilization, document projection, field manipulation, and memory management is a critical core competency for the MongoDB Certified Associate Developer Exam.

---

1. The Aggregation Pipeline Architecture

The Aggregation Framework operates on the concept of a data processing pipeline, analogous to Unix shell pipelines (cat data.txt | grep 'error' | awk '{print $2}' | sort).

+-----------------------------------------------------------------------------------------+
|                              MongoDB Aggregation Pipeline                               |
|                                                                                         |
|  [ Collection Docs ] ---> [ Stage 1: $match ] ---> [ Stage 2: $project ] ---> [ Result ]|
|                               (Filter)                  (Reshape)                       |
+-----------------------------------------------------------------------------------------+

Core Pipeline Characteristics

1. Document Stream Processing: Documents enter the pipeline sequentially from the source collection. Each stage receives a stream of BSON documents, applies its transformation or filter, and outputs a transformed stream of BSON documents to the subsequent stage.
2. Stage Independence & Composability: Each stage is an independent operation defined by a stage operator object (e.g., { $match: { ... } }, { $project: { ... } }). Stages can appear multiple times in a single pipeline, and their ordering dictates the precise sequence of execution.
3. Non-Destructive Execution: Unless an explicit materialization stage ($out or $merge) is appended at the very end of the pipeline, aggregation operations do not alter the underlying data in the source collection. They return a transient cursor over the computed result set.
4. Typing & BSON Expression Support: Aggregation expressions can perform mathematical computations, string manipulation, date arithmetic, conditional branching ($cond, $switch), and array filtering on document fields.

Basic Syntax in mongosh

db.collection.aggregate([
  { <stage 1> },
  { <stage 2> },
  { <stage 3> }
], { <pipeline options> });

---

2. Stage Ordering & The Pipeline Optimizer

The sequence of stages in an aggregation pipeline directly determines both the correctness of the final output and the runtime execution efficiency.

Why Stage Order Matters

Because each stage acts solely on the documents emitted by the immediately preceding stage, altering stage order changes the intermediate schema and dataset size:

• If a $project stage removes the status field, a subsequent $match stage filtering on { status: "ACTIVE" } will evaluate against null values and produce empty or incorrect results.
• If a $sort stage is placed before a $match stage, MongoDB must sort all documents in the collection before filtering out unwanted records, wasting substantial CPU and memory.

Automatic Pipeline Optimization

MongoDB includes an internal Aggregation Pipeline Optimizer that attempts to reorganize and coalesce stages before execution to optimize performance:

+---------------------------------------------------------------------------------+
|                        Pipeline Optimization Behaviors                          |
|                                                                                 |
|  1. $match Coalescing      : Adjacent $match stages combine into one $and clause|
|  2. $match Pushdown        : $match moves before $sort or $project if possible  |
|  3. $project Coalescing    : Adjacent $project stages merge into a single stage |
|  4. $limit / $skip Fusion  : Adjacent $sort and $limit combine to Top-N sort    |
+---------------------------------------------------------------------------------+

Best Practice Rule: Even though the optimizer performs stage pushdown where safe, developers should always explicitly place $match and $sort stages at the very beginning of the pipeline to guarantee optimal index utilization.

---

3. The $match Stage: Early Filtering & Index Utilization

The $match stage filters the document stream, allowing only documents matching the specified predicate to proceed to the next stage. It uses the exact same query syntax as the standard find() query filter.

Syntax

{ $match: { <query predicate> } }

Index Usage Mechanics

$match can leverage B-tree indexes to avoid full collection scans (COLLSCAN), but strict positional rules apply:

1. First Stage Execution: When $match is the first stage in an aggregation pipeline, MongoDB uses standard index scans (IXSCAN) just like db.collection.find().
2. Post-Transformation Invalidation: If $match appears after any stage that modifies the document structure (such as $project, $group, $unwind, or $addFields), it can no longer use indexes on the original collection because the stream now contains in-memory transformed documents.

// OPTIMAL: Uses compound index on { status: 1, orderDate: -1 }
db.orders.aggregate([
  {
    $match: {
      status: "DELIVERED",
      orderDate: { $gte: ISODate("2026-01-01T00:00:00Z") }
    }
  },
  {
    $project: {
      customerId: 1,
      totalAmount: 1,
      _id: 0
    }
  }
]);

// SUBOPTIMAL: $project precedes $match, preventing index utilization
db.orders.aggregate([
  {
    $project: {
      status: 1,
      orderDate: 1,
      totalAmount: 1
    }
  },
  {
    $match: {
      status: "DELIVERED" // Must perform in-memory scan of projected documents
    }
  }
]);

---

4. The $project Stage: Reshaping & Computing Projections

The $project stage reshapes each document in the stream by including existing fields, excluding fields, renaming fields, and creating entirely new computed fields using aggregation expressions.

Syntax

{
  $project: {
    <field1>: <1 | 0 | true | false | <expression>>,
    <field2>: <1 | 0 | true | false | <expression>>
  }
}

Inclusion vs. Exclusion Rules

1. Inclusion Mode: Specifying fields with 1 (or true) includes only those fields. All unspecified fields (except _id) are automatically suppressed from the output.
2. Exclusion Mode: Specifying fields with 0 (or false) excludes those specific fields, returning all other fields present in the input document.
3. The Mixed Projection Prohibition: You cannot mix 1 and 0 in the same $project stage, with exactly one exception: the _id field.

// VALID: Explicit inclusions + suppression of default _id
db.users.aggregate([
  {
    $project: {
      username: 1,
      email: 1,
      _id: 0 // Allowed: _id is the only field that can be excluded during inclusion
    }
  }
]);

// VALID: Explicit exclusions
db.users.aggregate([
  {
    $project: {
      passwordHash: 0,
      internalNotes: 0
    }
  }
]);

// INVALID: Throws MongoServerError: Cannot do inclusion on field email in exclusion projection
// db.users.aggregate([
//   { $project: { username: 1, passwordHash: 0 } }
// ]);

Computing Fields and Expressions in $project

$project is not limited to showing and hiding fields; it can compute new fields using aggregation expression operators:

db.orders.aggregate([
  {
    $project: {
      orderId: "$_id", // Field renaming
      customerUpper: { $toUpper: "$customerName" }, // String operator
      finalTotal: {
        $subtract: [
          { $add: [ "$subtotal", "$taxAmount" ] },
          { $ifNull: [ "$discount", 0 ] }
        ]
      }, // Arithmetic calculation
      isHighValue: { $gte: [ "$subtotal", 500 ] }, // Boolean expression
      _id: 0
    }
  }
]);

Exam Trap: In projection and aggregation expressions, referencing the value of an existing document field requires prefixing the field name with a dollar sign in quotes (e.g., "$subtotal"). Writing subtotal without $ treats the value as a literal string constant.

---

5. The $addFields and $set Stages: Preserving Document Shape

A common limitation of $project is that when you want to append one computed field to a document with 30 fields, you must explicitly list all 30 fields with 1 or else $project drops them.

To solve this, MongoDB provides $addFields (and its identical alias $set, introduced in MongoDB 4.2). These stages append new fields or overwrite existing fields while automatically preserving all other unspecified fields in the document.

Syntax Comparison

// Using $addFields
{ $addFields: { <newField1>: <expression1>, <existingFieldToOverwrite>: <expression2> } }

// Using $set (Exact functional alias to $addFields)
{ $set: { <newField1>: <expression1>, <existingFieldToOverwrite>: <expression2> } }

Practical Example: Adding Tax and Timestamp

db.invoices.aggregate([
  {
    $match: { status: "PENDING" }
  },
  {
    $set: {
      tax: { $multiply: [ "$subtotal", 0.0825 ] },
      totalDue: { $multiply: [ "$subtotal", 1.0825 ] },
      processedAt: "$$NOW" // System variable for current ISODate
    }
  }
]);

In this query, every original field in invoices (such as _id, invoiceNumber, customer, lineItems, paymentTerms) is retained in the output, while tax, totalDue, and processedAt are added.

Direct Comparison: $project vs. $addFields / $set

---

6. Memory Limits & allowDiskUse

Each individual stage in a MongoDB aggregation pipeline operates under a strict memory ceiling to prevent runaway queries from exhausting server RAM.

The 100 MB In-Memory Limit

By default, any individual pipeline stage is allocated a maximum of 100 MB of RAM for working state data. Stages that process documents in a streaming fashion (like $match and $project) process documents one at a time and rarely exceed this limit. However, blocking stages that must buffer multiple documents before emitting results—such as $sort (without an index), $group, and $bucket—can easily exceed 100 MB when processing large collections.

MongoServerError: PlanExecutor error during aggregation :: caused by :: 
Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. 
Aborting operation. Pass allowDiskUse: true to opt in.

Enabling Temporary Disk Spill: allowDiskUse: true

To allow memory-intensive pipeline stages to write temporary data files to the _tmp directory on the server disk, pass { allowDiskUse: true } in the aggregation options:

db.largeLogs.aggregate([
  {
    $match: { level: "ERROR" }
  },
  {
    $group: {
      _id: "$sourceIp",
      errorCount: { $sum: 1 },
      rawEvents: { $push: "$$ROOT" }
    }
  },
  {
    $sort: { errorCount: -1 }
  }
], {
  allowDiskUse: true // Permits pipeline stages to spill beyond 100 MB RAM to disk
});

Memory Limit Nuances & Atlas Restrictions

• In Atlas Shared Tier (M0, M2, M5) clusters, allowDiskUse: true is not permitted or severely restricted, making early filtering with $match and index-backed sorting critical.
• Document size limits remain unchanged: no individual output document emitted by any aggregation stage can ever exceed the 16 MB BSON document size limit.