4.3 Loops, Batch Processing & Rules

4.3 Loops, Batch Processing & Rules

Intermediate Exam Focus: Processing collections of records efficiently is a core competency tested on the Mendix Intermediate Developer certification. You must understand how the microflow loop container functions, how variable scope operates across loop iterations, how to construct batch processing chunking patterns to handle millions of records without running out of memory, why committing inside loops is a severe anti-pattern, and how declarative Business Rules differ from procedural microflow expressions.

Iterative processing is essential for business logic that must inspect, transform, or calculate values across multiple records. However, looping over large datasets improperly can quickly exhaust server resources, saturate database connection pools, and degrade application performance across an entire enterprise deployment.

---

Microflow Loop Structures and Iterator Mechanics

In Mendix Studio Pro, a Loop is a visual container that iterates sequentially over a collection of entities (List of [Entity]).

[Start] ──> [Retrieve $OrderList] ──> ╔══════════════════════════════════════════╗ ──> [End]
                                      ║ Loop: for each $IteratorOrder in List    ║
                                      ║   [Calculate Item Discount]              ║
                                      ║   [Add to $AccumulatorList]              ║
                                      ╚══════════════════════════════════════════╝

The Iterator Variable

When you configure a loop on an entity collection, Studio Pro automatically generates an Iterator Variable (e.g., $IteratorOrder or $IteratorCustomer). This variable represents the single, active entity instance being processed during the current cycle:

• The iterator variable is read-write: you can inspect its attributes, modify its values via Change Object, or evaluate associations.
• Scope Isolation within Loops: Any local variable declared inside the loop container (such as a temporary string variable or tax calculation decimal) is re-initialized on each iteration and destroyed when the iteration completes. It cannot be accessed outside the loop container.
• Accumulator Pattern: To preserve data across iterations, developers must declare an accumulator variable or collection (such as a List of [Entity] or a Decimal sum) before entering the loop. Inside the loop, activities append data to or modify this pre-existing outer variable.

The Concurrent Modification Constraint

Exam Trap: In Mendix, you must NEVER modify the structure of the list currently being looped over.

If you execute a Change List activity that adds or removes elements from the active $OrderList while iterating through $OrderList, the runtime will produce non-deterministic behavior, skip elements, or throw a java.util.ConcurrentModificationException.

The Correct Pattern: If you need to filter or collect items during a loop, create a separate empty list (e.g., $SelectedOrders) outside the loop, and use Change List (Add) to insert matching objects into that secondary list.

Enterprise Batch Processing: Managing Massive Datasets

When applications must process tens of thousands (or millions) of records—such as end-of-month financial reconciliations, bulk customer notifications, or nightly data warehouse synchronization—attempting to retrieve all records in a single database retrieve action will cause immediate failure:

1. JVM Heap Exhaustion: Loading 100,000 objects into memory will trigger a fatal OutOfMemoryError.
2. Database Lock Contention: A query touching hundreds of thousands of rows causes database lock escalations (converting row locks to table locks), freezing other concurrent users.
3. Transaction Timeouts: A microflow holding an open database transaction for minutes will trigger database connection pool timeouts and socket drops.

The Batch Chunking Pattern

To process massive datasets safely, enterprise developers implement the Batch Chunking Pattern. Instead of processing the whole dataset at once, the system processes records in discrete chunks (typically 500 to 1,000 objects):

Chunk 1 (1,000 items) ──> Process ──> Commit ──> Release Memory
Chunk 2 (1,000 items) ──> Process ──> Commit ──> Release Memory
Chunk 3 (1,000 items) ──> Process ──> Commit ──> Release Memory
... until 0 items returned.

The "Offset = 0" Trap in Status-Driven Processing

A critical architectural trap tested extensively on intermediate exams involves configuring Offset during chunked retrieves:

• Scenario A: Read-Only Processing (Offset must increment)

  • If you are reading records without modifying the criteria in the XPath constraint (for example, generating an export report of all historical orders):
  • Iteration 1: Limit = 1000, Offset = 0 (Processes records 1 to 1000)
  • Iteration 2: Limit = 1000, Offset = 1000 (Processes records 1001 to 2000)
  • Iteration 3: Limit = 1000, Offset = 2000 (Processes records 2001 to 3000)

• Scenario B: Status-Updating Processing (Offset MUST REMAIN 0!)

  • Suppose your XPath retrieve queries pending items: [Status = 'Pending'].
  • Inside the loop, your microflow updates each record's status to Status = 'Processed' and commits it.
  • What happens if you increment Offset?
    • In Iteration 1 (Offset = 0), you retrieve the first 1,000 pending items. You mark them Processed. They are no longer pending!
    • The remaining pending items automatically slide down to the front of the queue (index 0).
    • In Iteration 2, if you set Offset = 1000, the database skips the first 1,000 remaining pending items and retrieves items 1,001 through 2,000! You have just skipped 1,000 unprocessed records!

Golden Exam Rule: Whenever batch processing modifies records so they no longer match the XPath retrieve constraint, Offset must always remain 0 on every retrieve cycle.

---

Commit Optimization: Batch Commits vs. Commit Inside Loops

One of the most destructive anti-patterns in Mendix development is placing a Commit Object activity inside a loop.

The N+1 Database Commit Anti-Pattern

If a loop iterates over 2,000 objects and executes a Commit action on every iteration:

• The Mendix Runtime executes 2,000 separate SQL UPDATE queries across the network.
• The database engine performs 2,000 separate index updates and transaction log writes.
• Mendix triggers 2,000 separate entity event lifecycles (Before Commit / After Commit microflows).
• Execution time: typically 30 to 60 seconds.

The Optimized Batch Commit Pattern

Instead of committing iteratively:

1. Before entering the loop, initialize an empty collection: Create List -> $OrdersToCommit.
2. Inside the loop, configure the Change Object activity with Commit = No.
3. Add the changed object to $OrdersToCommit using Change List (Add).
4. After the loop finishes (or every 500 records), execute a single Commit List activity on $OrdersToCommit.

• Execution time: under 500 milliseconds (a 100x performance improvement!).

---

Business Rules vs. Microflow Expressions

Mendix Studio Pro provides two distinct mechanisms for evaluating conditional business logic: Microflow Expressions inside Decision splits and dedicated Business Rule documents.

What is a Business Rule?

A Business Rule is a specialized, declarative document in the project tree that encapsulates complex boolean validation logic. It accepts input parameters (entities, lists, or primitives) and evaluates to a single Boolean (true or false).

[Decision Split: Is Customer Eligible for Loan?]
       │
       ├──> Evaluates Business Rule: [Rule_Customer_LoanEligibility]
       │       ├── Input: $Customer
       │       ├── Input: $LoanAmount
       │       └── Returns: Boolean (True / False)

Technical Restrictions and Pure Functions

Business Rules are engineered as strictly pure functions without side effects. Inside a Business Rule, Studio Pro prohibits:

• Database retrieve actions
• Creating or deleting objects
• Modifying entity attributes
• Committing data to the database
• Calling web services or REST integrations
• Showing client-side messages or opening pages

A Business Rule can only contain Rule Decisions, Expressions, and calls to other Business Rules.

Comparison: Microflow Expressions vs. Business Rules