5.1 Index Types & Storage Mechanics

Index Types & Storage Mechanics

Exam Focus: The MongoDB Certified Associate Developer Exam tests the fundamental storage engine mechanics of indexing under WiredTiger, the immutable constraints of the default _id index, single-field index directionality and bidirectional traversal, dot notation indexing for subdocument fields, index prefix matching, and index management commands (createIndex, getIndexes, dropIndex).

---

The Fundamental Role of Indexes in MongoDB

Without indexes, MongoDB must execute a Collection Scan (COLLSCAN) to satisfy any read query. In a collection scan, the storage engine sequentially reads every BSON document stored in the collection's data files from disk into memory, evaluating the query predicate against each document one by one. For a collection containing millions of documents, a COLLSCAN requires linear $O(N)$ time complexity, saturates storage I/O bandwidth, evicts cached working sets from the WiredTiger cache, and causes severe application latency.

An Index is a specialized, ordered data structure that stores a small portion of the collection's data set in an easily traversable form. The index holds the values of specific fields ordered by the value of the field, along with a pointer (the RecordId) directly referencing the document's physical location on disk. With an index, MongoDB performs an Index Scan (IXSCAN), traversing the index in logarithmic $O(\log N)$ time to locate matching entries and directly retrieving only the relevant documents.

---

WiredTiger B-Tree Index Architecture

In MongoDB's default WiredTiger storage engine, indexes are implemented as balanced search trees (B-trees). A B-tree is a self-balancing tree data structure that maintains sorted data and permits sequential access, searches, insertions, and deletions in logarithmic time.

Anatomical Layers of a WiredTiger B-Tree

1. Root Page (Node): The topmost page of the B-tree. It contains key ranges and memory pointers directing traversals to intermediate branch pages.
2. Internal Pages (Branch Nodes): Intermediate routing nodes that store separator keys and child page pointers. They partition the key space into progressively narrower ranges.
3. Leaf Pages (Leaf Nodes): The bottom layer of the tree. Leaf pages store the actual indexed key values in sorted ascending or descending order, paired with 64-bit internal RecordIds. The RecordId serves as the direct physical pointer to the document's uncompressed location in the collection data file.

                         +-----------------------+
                         |       ROOT PAGE       |
                         |  [Keys: 100 | 500]    |
                         +-----------+-----------+
                                    / \ 
             +---------------------+   +---------------------+
             | INTERNAL PAGE A     |   | INTERNAL PAGE B     |
             | [Keys: 25 | 50]     |   | [Keys: 650 | 800]   |
             +----------+----------+   +----------+----------+
                       / \                        / \
      +---------------+   +---------------+      ... ...
      |  LEAF PAGE 1  |   |  LEAF PAGE 2  |
      | Key: 10 -> R1 |   | Key: 30 -> R3 |
      | Key: 20 -> R2 |   | Key: 45 -> R4 |
      +---------------+   +---------------+
              |                   |
              v                   v
      [ Document R1 ]     [ Document R3 ]  (Collection Data File)

Logarithmic Search Efficiency

Because WiredTiger B-tree pages maintain high fan-out factors (often hundreds of keys per page), a B-tree indexing tens of millions of records typically requires a depth of only 3 to 4 page traversals from the root to the leaf. This reduces the disk page lookups from millions of document reads down to 3 or 4 memory-cached page evaluations.

---

The Default _id Index

Every MongoDB collection automatically possesses a unique index on the _id field. MongoDB creates this index during collection initialization.

Invariant Rules of the _id Index

• Automatic Creation: You do not need to explicitly call createIndex({ _id: 1 }). The server automatically provisions the index named _id_.
• Strict Uniqueness: The _id index enforces uniqueness across the collection, preventing duplicate primary keys.
• Immutability & Non-Droppable: The _id index cannot be dropped, renamed, or modified. Any attempt to invoke db.collection.dropIndex("_id_") or db.collection.dropIndex({ _id: 1 }) results in an immediate runtime error:

// Attempting to drop the default _id index in mongosh:
db.users.dropIndex("_id_");
// Throws MongoServerError: cannot drop _id index

• Capped Collections Exception: Even capped collections maintain the default _id index in modern MongoDB versions.

---

Single-Field Indexes

A Single-Field Index holds references to a single field within documents in a collection. It is the most fundamental user-defined index type.

Creation Syntax

// Create an ascending single-field index on 'email'
db.users.createIndex({ email: 1 });

// Create a descending single-field index on 'score'
db.leaderboard.createIndex({ score: -1 });

• The value 1 specifies an ascending index order (sorting values from lowest to highest: $A \rightarrow Z$, $1 \rightarrow 100$).
• The value -1 specifies a descending index order (sorting values from highest to lowest: $Z \rightarrow A$, $100 \rightarrow 1$).

Bidirectional Index Traversal for Single-Field Indexes

A critical concept on the certification exam is Bidirectional Index Traversal. For single-field indexes, the specified sort direction (1 vs -1) does not matter for single-field queries and sorts. Because the B-tree can be traversed forward from the beginning or backward from the end with identical computational efficiency, an ascending index on { age: 1 } fully supports both ascending and descending sorts:

// Given an index on { age: 1 }:

// 1. Forward Traversal: Scans index from smallest to largest age
db.users.find().sort({ age: 1 });   // Supported by { age: 1 } via forward scan

// 2. Backward Traversal: Scans index in reverse from largest to smallest age
db.users.find().sort({ age: -1 });  // Supported by { age: 1 } via backward scan

[!NOTE] Index key directionality only becomes critical when designing Compound Indexes spanning two or more fields. For any single-field index, { field: 1 } and { field: -1 } are functionally equivalent for query filtering and sorting.

---

Indexing Embedded Subdocuments & Dot Notation

MongoDB supports indexing fields inside nested subdocuments as well as entire embedded subdocuments.

1. Indexing Specific Embedded Fields via Dot Notation (Recommended)

To index a scalar field located within an embedded subdocument, use standard dot notation enclosed in quotes:

// Document structure: { _id: 1, name: "Acme", address: { city: "Austin", zip: "78701" } }

// Create an index specifically on the nested 'zip' field
db.companies.createIndex({ "address.zip": 1 });

// Queries utilizing this index:
db.companies.find({ "address.zip": "78701" });
db.companies.find({ "address.zip": { $gte: "78700", $lte: "78799" } });

This creates a lightweight index containing only the string values of address.zip.

2. Indexing an Entire Subdocument (Caution)

MongoDB allows indexing the entire subdocument object:

db.companies.createIndex({ address: 1 });

[!WARNING] Subdocument Indexing Pitfall: When an entire subdocument is indexed ({ address: 1 }), queries will only match if the query subdocument has the exact same field order and identical fields. A query for { address: { zip: "78701", city: "Austin" } } will NOT match an on-disk document stored as { address: { city: "Austin", zip: "78701" } }. Always index specific embedded fields using dot notation rather than full subdocuments.

---

Index Prefix Matching Principle

An Index Prefix is a contiguous subset of index keys starting from the very first field of an index definition.

For a compound index defined as { a: 1, b: 1, c: 1 }, the valid index prefixes are:

1. { a: 1 }
2. { a: 1, b: 1 }
3. { a: 1, b: 1, c: 1 } (the complete index)

Prefix Reuse & Eliminating Redundant Indexes

If a collection has an index on { lastName: 1, firstName: 1 }, MongoDB can use this compound index to satisfy queries on:

• Both lastName and firstName: find({ lastName: "Smith", firstName: "John" })
• lastName alone: find({ lastName: "Smith" })

Therefore, creating a separate single-field index on { lastName: 1 } is completely redundant. Maintaining redundant indexes wastes RAM in the WiredTiger cache, consumes excess disk storage, and degrades write performance because every insert, update, and delete must update multiple identical B-tree paths.

[!IMPORTANT] Non-prefix subsets—such as querying on { firstName: 1 } alone—cannot use the index prefix of { lastName: 1, firstName: 1 } because the primary sort in the B-tree is organized by lastName.

---

Index Administration & Lifecycle Management

MongoDB provides shell methods and database commands to inspect, create, rename, and drop indexes.

1. Listing Existing Indexes: getIndexes()

db.customers.getIndexes();

Returns an array of index specification documents:

[
  {
    "v": 2,
    "key": { "_id": 1 },
    "name": "_id_"
  },
  {
    "v": 2,
    "key": { "email": 1 },
    "name": "email_1"
  }
]

2. Custom Index Naming

By default, MongoDB generates index names by concatenating field names and sort orders (e.g., email_1_status_-1). You can provide a custom identifier using the name option:

db.customers.createIndex(
  { email: 1, status: -1 },
  { name: "idx_active_customer_lookup" }
);

3. Dropping Indexes: dropIndex() and dropIndexes()

Indexes consume memory and I/O. Unused indexes should be dropped promptly:

// Drop by index name string
db.customers.dropIndex("idx_active_customer_lookup");

// Drop by index key specification
db.customers.dropIndex({ email: 1, status: -1 });

// Drop ALL user-created indexes on the collection (leaves _id_ intact)
db.customers.dropIndexes();

4. Modern Hybrid Index Builds

In modern MongoDB versions, createIndex() uses a hybrid index build process. Hybrid builds obtain a quick read/write lock only at the start and end of the build, allowing concurrent reads and writes to proceed unhindered during the main B-tree construction phase.

---

Counting the Indexes on a Collection

Exam objective 3.4 asks you to identify how many indexes exist for a collection, and it is nearly always an arithmetic trap rather than a syntax question.

db.collection.getIndexes() returns an array of index specification documents, one per index, so db.collection.getIndexes().length is the count:

db.products.getIndexes()
[
  { v: 2, key: { _id: 1 },              name: "_id_" },
  { v: 2, key: { sku: 1 },              name: "sku_1" },
  { v: 2, key: { category: 1, price: -1 }, name: "category_1_price_-1" }
]
// -> 3 indexes

The _id Index Always Counts

Every collection receives a unique index on _id automatically at creation, named _id_. You did not create it, you cannot drop it, and it counts toward the total.

So when a scenario says "a developer creates a single-field index on sku and a compound index on category and price," the answer is three indexes, not two. Two is the distractor, and it catches candidates who count only the createIndex() calls in the stem.

A compound index counts as one index no matter how many fields it spans — the three-field index { a: 1, b: 1, c: 1 } is one index, not three. Note also that db.collection.dropIndexes() removes all indexes except _id_, so a collection can never drop below one index.