1.3 Distributed Architecture Fundamentals: Replica Sets & Sharded Clusters

Distributed Architecture Fundamentals: Replica Sets & Sharded Clusters

Exam Focus: The Associate Developer Exam evaluates distributed MongoDB architectures across two domains: High Availability via Replica Sets (Primary/Secondary roles, majority election quorums, arbiter tradeoffs, oplog idempotency, rollback mechanics) and Horizontal Scalability via Sharded Clusters (mongos routing mechanics, Config Server metadata, shard key selection principles, and targeted vs. scatter-gather queries).

---

High Availability: Replica Set Architecture & Member Topologies

A Replica Set is a group of mongod instances that maintain the same data set, providing redundancy, fault tolerance, and high availability. A production replica set contains exactly one Primary node and one or more Secondary nodes.

Node Roles & Topologies

• Primary Node: The single member in the replica set that receives and executes all write operations. The primary records all state modifications into its local operation log (oplog). By default, clients also direct all read operations to the primary, guaranteeing strict read-your-writes consistency.
• Secondary Nodes: Replicate the primary's oplog asynchronously and apply the operations to their local datasets. Secondaries can serve read operations when client drivers configure non-primary read preferences (secondary, secondaryPreferred, nearest).
• Arbiter Node: A lightweight mongod instance configured solely to participate in election voting. An arbiter:
  • Holds 1 vote in elections.
  • Does NOT hold a copy of the dataset (zero data storage).
  • Cannot become a primary.
  • Exists purely to maintain an odd number of voting members in clusters where deploying an additional full data-bearing secondary is cost-prohibitive.
• Specialized Secondary Configurations:
  • Priority 0 Member (priority: 0): Cannot become primary under any circumstances. Can vote in elections and serve read operations. Useful for cross-region disaster recovery standby nodes.
  • Hidden Member (hidden: true, priority: 0): Invisible to client drivers and cannot serve client read requests. Votes in elections. Ideal for dedicated analytics reporting or running heavy backup processes (mongodump).
  • Delayed Secondary (secondaryDelaySecs: 3600): Maintains a deliberate time-delayed copy of the dataset (e.g., 1 hour behind). Acts as insurance against human error (such as accidentally dropping a production database).

---

Consensus, Elections & Automated Failover Mechanics

MongoDB replica sets use a Raft-inspired consensus algorithm to elect a new primary automatically when the existing primary becomes unreachable.

The Strict Majority Quorum Rule

To elect or maintain a primary, a replica set must assemble a strict majority of its total configured voting members. The majority threshold is calculated as:

$\text{Majority Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$

Where $N$ is the total number of voting members in the replica set configuration.

[!IMPORTANT] Why an Odd Number of Voting Nodes is Required: Notice that a 4-node cluster requires 3 votes for a majority and can tolerate only 1 failure—the exact same fault tolerance as a 3-node cluster! If a network partition splits a 4-node cluster into two 2-node halves ($2 + 2$), neither half can achieve the required 3-node majority, causing an election deadlock. Therefore, voting topologies should always contain an odd number of voting members (3, 5, 7).

Failover Timeline

1. Heartbeat Probing: All members send periodic heartbeat pings to every other member every 2 seconds.
2. Election Trigger: If secondaries do not receive a heartbeat response from the primary within the electionTimeoutMillis window (default: 10 seconds), an election is triggered.
3. Candidate Nomination: An eligible secondary nominates itself if it can reach a majority of nodes and possesses the most up-to-date oplog entry among reachable peers.
4. Vote Granting & Promotion: Once the candidate receives votes from a strict majority ($> 50%$), it transitions to Primary and begins accepting client writes. Total election failover typically completes in under 5 to 10 seconds.

---

The Replication Oplog & Idempotency

The Oplog (oplog.rs) is a specialized capped collection located in the local database of every replica set member.

The Principle of Idempotency

Every write operation applied by the primary is translated into an idempotent oplog entry. An operation is idempotent if applying it once produces the exact same system state as applying it multiple times consecutively.

• Non-idempotent Client Command: db.inventory.updateOne({ _id: 101 }, { $inc: { stock: 5 } })
• Idempotent Oplog Entry Recorded: The primary resolves the math and writes an absolute $set operation to the oplog: { op: "u", ns: "store.inventory", o2: { _id: 101 }, o: { $set: { stock: 25 } } }

Because oplog entries are idempotent, secondaries can replay batches of oplog operations concurrently or re-fetch historical segments during network recovery without risk of data corruption or cumulative skew.

---

Uncommitted Writes & Rollback Mechanics

When a primary node experiences a network partition or hardware crash while processing writes, some writes may have been written to its local oplog without being replicated to a majority of secondaries.

1. Primary Deposition: The remaining secondaries form a majority quorum and elect a new primary.
2. State Divergence: The new primary accepts writes, advancing its oplog timeline.
3. Node Rejoining: When the old primary recovers and rejoins the cluster, it discovers it is now a secondary and its oplog timeline diverged from the elected primary.
4. Rollback Execution: The rejoining node rolls back its uncommitted local writes to match the current primary's timeline. It extracts the orphaned documents and writes them to a rollback BSON file in the diagnostic.data/rollback/ directory for manual administrator inspection.

[!TIP] Applications prevent rollbacks completely by issuing critical writes with a write concern of w: "majority" ({ w: "majority", wtimeout: 5000 }), which guarantees that write acknowledgement is delayed until a strict majority of replica set nodes have committed the data to their oplogs.

---

Horizontal Scalability: Sharded Cluster Architecture

When a dataset grows beyond the storage capacity, RAM caching limits, or I/O throughput of a single physical replica set, MongoDB scales horizontally using Sharding.

The Three Core Components

1. mongos (Query Router): A stateless routing process that acts as the interface between client applications and the sharded cluster. Applications connect to mongos instances exactly as they would to a standalone mongod. The mongos reads incoming queries, consults the Config Server metadata cache to locate the appropriate shard(s), routes the operations, and aggregates responses.
2. Config Servers: A dedicated 3-member replica set (configsvr) that stores the cluster's master catalog and routing table. It records the mapping of chunks to shards and manages cluster-wide lock transactions.
3. Shards: Independent Replica Sets that store a subset of the sharded data. In production, every shard must be deployed as a multi-node replica set to ensure high availability for its data partition.

---

Shard Key Engineering: Cardinality, Frequency & Monotonicity

A Shard Key is an immutable field or compound fields indexed within the collection that dictates how MongoDB partitions documents into Chunks across shards.

1. Key Properties of Effective Shard Keys

• High Cardinality (Number of Distinct Values): The shard key must have a vast number of unique values. A boolean field (is_active) has a cardinality of 2; a collection partitioned on is_active can never be split into more than 2 chunks, rendering horizontal scaling impossible.
• Low Frequency (Even Distribution of Values): If 80% of all documents share the same shard key value (e.g., country: "US"), the chunk holding "US" will grow uncontrollably into an indivisible Jumbo Chunk, bottlenecking a single shard.
• Non-Monotonic Write Distribution: Monotonically increasing keys (such as ObjectId, auto-incrementing integers, or timestamps) route all new inserts to the chunk with the maximum range [MaxKey]. This causes write hot-spotting, directing 100% of cluster write throughput to a single shard while other shards sit idle.

2. Range-Based vs. Hashed Sharding

To balance query targeting against write distribution, MongoDB offers two partitioning strategies:

• Range-Based Sharding: Documents are partitioned into contiguous ranges based on the raw shard key values. Ideal for optimizing range queries ($gte, $lte), but vulnerable to insert hot-spotting with monotonic keys.
• Hashed Sharding: MongoDB computes an MD5 hash of the shard key field ({ user_id: "hashed" }) and partitions data based on hash ranges. Consecutive sequential inserts produce completely random hashes, distributing writes evenly across all shards.

3. Targeted Queries vs. Scatter-Gather Queries

• Targeted Query (Single-Shard): When a query includes the shard key (e.g., db.users.find({ user_id: 10842 })), mongos inspects its cached chunk mapping and routes the query directly to the single shard holding that chunk. This yields sub-millisecond response times.
• Scatter-Gather Query (Multi-Shard): When a query omits the shard key (e.g., db.users.find({ email: "alex@example.com" })), mongos has no routing information. It must broadcast the query to every shard in the cluster, wait for all shards to respond, and merge the result sets before returning them to the client. Scatter-gather operations introduce high network overhead and latency spikes.

---

Distributed Administration & Diagnostics in mongosh

// --- REPLICA SET MANAGEMENT ---

// 1. Inspect replica set health, member states, and replication lag
rs.status();

// 2. View current replica set configuration document
rs.conf();

// 3. Step down the current primary for maintenance (forces an immediate election)
rs.stepDown(60); // Step down for 60 seconds

// 4. Check node identity and replication role
db.hello();


// --- SHARDED CLUSTER MANAGEMENT ---

// 1. Check sharded cluster status, shard list, databases, and chunk distribution
sh.status();

// 2. Enable sharding on a target database
sh.enableSharding("ecommerce_db");

// 3. Create a hashed index on the shard key field
use ecommerce_db;
db.orders.createIndex({ order_id: "hashed" });

// 4. Shard the collection using the hashed shard key
sh.shardCollection("ecommerce_db.orders", { order_id: "hashed" });