7.2 Batch Job Architecture: Input, Process, and Complete Phases

Batch Job Architecture: Load and Dispatch, Process, and On Complete Phases

When integrations must synchronize hundreds of thousands or millions of records between enterprise systems (such as syncing ERP accounts to Salesforce or processing overnight billing feeds), synchronous in-memory iteration risks JVM OutOfMemoryError failures and network timeout crashes. Mule 4 solves large-scale data synchronization through its Batch Processing Architecture (<batch:job>).

Mule Batch Jobs provide automated payload splitting, disk-backed persistent queueing, multi-threaded parallel execution across blocks of records, fine-grained error tolerance, and asynchronous execution handoffs. Mastering the anatomy of a Batch Job and its three distinct phases is a cornerstone of the Developer I certification.

---

1. High-Level Batch Architecture & Lifecycle

A Batch Job splits large collections or streaming datasets into individual records, stores them in persistent queues on disk, processes them asynchronously through a series of Batch Steps, and finally generates a comprehensive execution report in the On Complete phase.

+---------------------------------------------------------------------------------------------------------+
|                                      MULE 4 BATCH JOB ARCHITECTURE                                      |
|                                                                                                         |
|  [Calling Flow]                                                                                         |
|        |                                                                                                |
|        v (Inbound Collection: 10,000 Records)                                                           |
|  +------------+                                                                                         |
|  | <batch:job>| ---> [ASYNC HANDOFF] ---> Calling flow immediately receives BatchJobInstance and continues! |
|  +------------+                                                                                         |
|        |                                                                                                |
|        v                                                                                                |
|  =====================================================================================================  |
|  PHASE 1: LOAD AND DISPATCH PHASE (Implicit; no XML element)                                            |
|  - Splits collection into 10,000 individual record objects                                              |
|  - Creates persistent disk queue (.mule/.queue) and serializes records into blocks (default: 100)       |
|  =====================================================================================================  |
|        |                                                                                                |
|        v                                                                                                |
|  =====================================================================================================  |
|  PHASE 2: PROCESS PHASE (<batch:process-records>)                                                       |
|  - Worker threads pull blocks from queue and process records through sequential Batch Steps             |
|                                                                                                         |
|    +-----------------------------------------------------------------------------------------------+    |
|    | <batch:step name="Step_1_Validate_And_Enrich"> (Worker Threads execute concurrently)          |    |
|    |   - Filters / transforms record; sets Record-Level Variables (vars.recordStatus = 'VALID')    |    |
|    +-----------------------------------------------------------------------------------------------+    |
|                                            | (Queued to next step)                                      |
|                                            v                                                            |
|    +-----------------------------------------------------------------------------------------------+    |
|    | <batch:step name="Step_2_Upsert_To_Target">                                                   |    |
|    |   - Calls Target API; routes failed records based on acceptPolicy                             |    |
|    +-----------------------------------------------------------------------------------------------+    |
|  =====================================================================================================  |
|        |                                                                                                |
|        v                                                                                                |
|  =====================================================================================================  |
|  PHASE 3: ON COMPLETE PHASE (<batch:on-complete>)                                                       |
|  - Executes ONCE when all records finish processing (or max failures reached)                           |
|  - payload = BatchJobResult Object (totalRecords: 10000, successfulRecords: 9950, failedRecords: 50)   |
|  =====================================================================================================  |
+---------------------------------------------------------------------------------------------------------+

---

2. Asynchronous Flow Handoff

A critical behavioral rule of <batch:job> is its asynchronous handoff model:

1. When a flow encounters <batch:job>, the Mule runtime initializes a new BatchJobInstance.
2. The collection is accepted into the Load and Dispatch phase, and the runtime immediately returns execution control to the calling flow.
3. The calling flow proceeds to its next component without waiting for the batch job's steps or On Complete phase to finish.
4. In the calling flow, the payload immediately after <batch:job> is a BatchJobInstance metadata object (containing batchJobInstanceId, status, creation timestamp).

<flow name="trigger-batch-flow">
    <http:listener config-ref="HTTP_Listener_config" path="/start-sync" doc:name="Start Sync" />
    <db:select config-ref="DB_Config" doc:name="Fetch 50k Accounts">
        <db:sql>SELECT * FROM raw_accounts</db:sql>
    </db:select>
    
    <!-- Asynchronous handoff occurs here -->
    <batch:job jobName="Account_Sync_Batch_Job" doc:name="Account Batch Job">
        <batch:process-records>
            <batch:step name="Process_Accounts">
                <!-- Step Processors -->
            </batch:step>
        </batch:process-records>
    </batch:job>
    
    <!-- This logger executes IMMEDIATELY while the batch job runs in the background! -->
    <logger level="INFO" message="#['Batch job started with instance ID: ' ++ payload.batchJobInstanceId]" />
    
    <!-- HTTP Listener returns 200 OK to caller right away -->
    <set-payload value="#[{ status: 'ACCEPTED', jobId: payload.batchJobInstanceId }]" doc:name="Return Ack" />
</flow>

---

3. The Three Lifecycle Phases in Depth

Phase 1: Load and Dispatch Phase

Mule 4 has no batch:input element — the batch job is a scope inside a normal flow, and this first phase is implicit.

• Payload Splitting: Takes the incoming collection or streaming CursorProvider and splits it into individual record items.
• Queue Creation: Creates a temporary, persistent queue on disk (managed under the .mule application directory).
• Serialization & Block Creation: Serializes records and stores them in blocks governed by the blockSize attribute.
• Flow Variable Snapshotting: Takes a snapshot of all flow variables existing in the Mule event prior to the batch job. These variables are attached as initial Record Variables for each record.

Phase 2: Process Phase (<batch:process-records>)

• The only mandatory phase in a Batch Job.
• Contains one or more <batch:step> elements executed in strict sequential order (Step 1 -> Step 2 -> Step N).
• Concurrent Worker Processing: Within each step, multiple worker threads pull record blocks from the queue and process individual records concurrently.
• Records that complete Step 1 are placed back onto the queue for Step 2. Records never wait for the entire dataset to finish Step 1 before beginning Step 2; records advance as blocks complete.

Phase 3: On Complete Phase (<batch:on-complete>)

• Optional phase executed exactly once after all records have finished all batch steps (or when maxFailedRecords is reached).
• Receives a BatchJobResult object in payload containing holistic statistics about the batch run.
• Used for sending completion emails, publishing Slack alerts, updating audit logs, or cleaning up temporary resources.

---

4. Batch Block Size (blockSize) & Persistent Queues

The blockSize attribute on <batch:job> (default: 100) controls how records are chunked for I/O operations and thread scheduling:

<batch:job jobName="Optimized_Batch_Job" blockSize="200" maxFailedRecords="-1">
    <!-- ... -->
</batch:job>

Why Block Sizing Matters:

• Queue I/O Performance: Reading and writing individual records to disk queues creates excessive disk I/O and locking overhead. Grouping records into blocks of 100 (or 200) allows batch writes to disk.
• Thread Utilization: Mule worker threads pick up an entire block from the queue and process its records before fetching the next block.
• Memory Safety: By buffering only a limited number of blocks in memory at any given time, Mule can process multi-gigabyte datasets on small runtime profiles (such as 0.1 or 0.2 vCore CloudHub workers) without exhausting JVM heap memory.

---

5. Variable Scoping: Record Variables vs. Flow Variables

Understanding variable scope within a Batch Job is one of the most frequently tested areas on the MuleSoft Developer I exam.

+-----------------------------------------------------------------------------------------+
|                                 BATCH VARIABLE SCOPING                                  |
|                                                                                         |
|  1. BEFORE BATCH JOB (Calling Flow):                                                    |
|     vars.globalConfig = "PROD"                                                          |
|     vars.counter      = 0                                                               |
|                                            |                                            |
|                                            v                                            |
|  2. INSIDE BATCH STEP (Record-Level Scope):                                             |
|     - Each record receives its OWN COPY of vars.globalConfig and vars.counter           |
|     - Inside Step 1: <set-variable variableName="recordTax" value="#[payload.amt * 0.1]"/>|
|     - vars.recordTax is bound ONLY to this individual record                            |
|     - vars.recordTax travels with this record into Step 2 and Step 3                    |
|                                            |                                            |
|                                            v                                            |
|  3. IN ON COMPLETE PHASE (<batch:on-complete>):                                         |
|     - payload is BatchJobResult                                                         |
|     - Record-level variables (vars.recordTax) are NOT accessible!                       |
|                                            |                                            |
|                                            v                                            |
|  4. IN CALLING FLOW (After Batch Handoff):                                              |
|     - vars.counter remains 0 (mutations inside batch steps NEVER propagate back!)       |
|     - vars.recordTax DOES NOT EXIST in the calling flow                                 |
+-----------------------------------------------------------------------------------------+

Rules of Batch Variable Scoping:

1. Inbound Inheritance: All flow variables that exist prior to <batch:job> are copied into every individual record as initial record variables.
2. Record Isolation: Inside a <batch:step>, executing <set-variable> creates or modifies a Record-Level Variable. It is attached strictly to that specific record.
3. Step-to-Step Propagation: Record variables set in Step 1 are accessible in Step 2, Step 3, and inside <batch:aggregator> for that same record.
4. No Reverse Propagation: Record variable mutations never affect other records, never modify variables in the calling flow, and are not available in <batch:on-complete>.

---

6. The BatchJobResult Object in On Complete

Inside the <batch:on-complete> phase, payload is an instance of BatchJobResult. It exposes the following read-only attributes via DataWeave:

Example: Generating an Execution Report in <batch:on-complete>

<batch:on-complete>
    <ee:transform doc:name="Create Summary Payload">
        <ee:message>
            <ee:set-payload><![CDATA[%dw 2.0
output application/json
---
{
    jobId: payload.batchJobInstanceId,
    summary: {
        total: payload.totalRecords,
        successful: payload.successfulRecords,
        failed: payload.failedRecords,
        durationSec: payload.elapsedTimeInMillis / 1000
    },
    successRate: ((payload.successfulRecords / payload.totalRecords) * 100) ++ "%"
}]]></ee:set-payload>
        </ee:message>
    </ee:transform>
    <logger level="INFO" message="#['Batch Execution Finished: ' ++ write(payload, 'application/json')]" />
</batch:on-complete>

---

7. Exam Watch: Core Batch Architecture Scenarios

[!IMPORTANT] Asynchronous Calling Flow Output When a question asks what payload is returned to the client immediately after triggering a flow with a <batch:job>, the answer is a BatchJobInstance summary object (or whatever subsequent components in the calling flow set), NOT the results of the batch processing.

[!WARNING] Record Variables Do NOT Reach On Complete You cannot access individual record data or record variables inside <batch:on-complete>. payload in On Complete is strictly the BatchJobResult statistical object.

[!TIP] Block Size vs Aggregator Size Do not confuse blockSize (a job-level attribute governing queue threading, default 100) with <batch:aggregator size="N"> (a step-level construct grouping records into arrays for external bulk operations).