7.3 Batch Step Accept Expression, Failure Handling & Batch Aggregator

Batch Step Accept Expression, Failure Handling & Batch Aggregator

In complex enterprise batch integrations, not all records follow an identical processing path. Some records require specialized enrichment, others fail validation and must be routed to dead-letter queues, and high-volume target systems require records to be aggregated into bulk arrays rather than processed individually.

Mule 4 provides sophisticated record routing and bulk aggregation capabilities within the Process Phase of a Batch Job. Mastering acceptExpression, acceptPolicy, maxFailedRecords, and the <batch:aggregator> scope is crucial for designing fault-tolerant ETL pipelines and excelling on the MuleSoft Developer I exam.

---

1. Batch Step Execution & Routing Mechanics

The Process Phase contains one or more <batch:step> elements. Records transition through batch steps in sequential order. However, within any single batch step, multiple records are processed concurrently across worker threads.

Each batch step evaluates two gating criteria before allowing a record to execute its child message processors:

1. acceptExpression (DataWeave Filter): Evaluates whether the record meets specific business criteria.
2. acceptPolicy (Failure Filter): Evaluates whether the record's current success/failure state qualifies it for processing.

+-----------------------------------------------------------------------------------------+
|                              BATCH STEP GATING LOGIC                                    |
|                                                                                         |
|   Incoming Record from Queue                                                            |
|               |                                                                         |
|               v                                                                         |
|   +-----------------------+                                                             |
|   | Check acceptPolicy    | ---> Record failed previously, but policy is NO_FAILURES?   |
|   +-----------------------+      |                                                      |
|               | (PASS)           +---> [SKIP STEP] ---> Forward to Next Step Queue      |
|               v                                                                         |
|   +-----------------------+                                                             |
|   | Check acceptExpression| ---> Expression evaluates to false?                         |
|   +-----------------------+      |                                                      |
|               | (TRUE)           +---> [SKIP STEP] ---> Forward to Next Step Queue      |
|               v                                                                         |
|   +---------------------------------------------------------------------------------+   |
|   | Execute Step Processors (Transform, HTTP Request, Database, Aggregator)         |   |
|   +---------------------------------------------------------------------------------+   |
|               |                                                                         |
|               +---> Success: Record remains marked SUCCESSFUL                           |
|               +---> Unhandled Exception: Record marked FAILED                           |
+-----------------------------------------------------------------------------------------+

---

2. Filtering Records with acceptExpression

The acceptExpression attribute contains a DataWeave boolean expression evaluated against each individual record.

• If the expression evaluates to true, the record enters the step and executes its processors.
• If the expression evaluates to false, the record skips the step completely and moves directly to the queue for the next batch step.
• Skipping a step does not mark the record as failed; it simply bypasses execution.

<!-- Step executes ONLY for records where tier is PLATINUM and balance > 1000 -->
<batch:step name="Process_VIP_Customers" 
            acceptExpression="#[payload.tier == 'PLATINUM' and (payload.balance default 0) &gt; 1000]">
    <http:request method="POST" config-ref="VIP_Service_Config" path="/vip-sync">
        <http:body><![CDATA[#[payload]]]></http:body>
    </http:request>
</batch:step>

---

3. Failure Handling with acceptPolicy

When a message processor inside a <batch:step> throws an unhandled exception, Mule catches the exception, attaches error metadata to the record, and flags the record as FAILED. By default, failed records are skipped by all subsequent steps to prevent cascading corruption.

The acceptPolicy attribute controls which records a step will process based on their failure status:

XML Configuration: Dead-Letter Queueing for Failed Records

<batch:job jobName="Resilient_Order_Sync" maxFailedRecords="-1">
    <batch:process-records>
        <!-- Step 1: Process only healthy records -->
        <batch:step name="Step1_Charge_Credit_Card" acceptPolicy="NO_FAILURES">
            <!-- If this HTTP call fails, record is marked FAILED -->
            <http:request method="POST" config-ref="Payment_Gateway_Config" path="/charge" />
        </batch:step>
        
        <!-- Step 2: Process only healthy records that succeeded in Step 1 -->
        <batch:step name="Step2_Fulfill_Order" acceptPolicy="NO_FAILURES">
            <jms:publish config-ref="JMS_Config" destination="orders.fulfillment" />
        </batch:step>
        
        <!-- Step 3: DEAD-LETTER STEP - Executes ONLY for records that failed Step 1 or Step 2 -->
        <batch:step name="Step3_Handle_Failures" acceptPolicy="ONLY_FAILURES">
            <logger level="ERROR" message="#['Record failed processing: ' ++ payload.orderId ++ ' Error: ' ++ error.description]" />
            <!-- Write failed record to dead-letter database table -->
            <db:insert config-ref="Database_Config" doc:name="Log to DLQ Table">
                <db:sql><![CDATA[
                    INSERT INTO failed_orders (order_id, payload_json, error_message) 
                    VALUES (:orderId, :rawPayload, :errMsg)
                ]]></db:sql>
                <db:input-parameters><![CDATA[#[
                    {
                        orderId: payload.orderId,
                        rawPayload: write(payload, 'application/json'),
                        errMsg: error.description default 'Unknown Processing Error'
                    }
                ]]]></db:input-parameters>
            </db:insert>
        </batch:step>
    </batch:process-records>
</batch:job>

---

4. Job Failure Thresholds: maxFailedRecords

The maxFailedRecords attribute on <batch:job> defines the failure threshold before the entire batch job is aborted:

<batch:job jobName="Batch_With_Threshold" maxFailedRecords="50">
    <!-- ... -->
</batch:job>

Threshold Behaviors:

• maxFailedRecords="-1" (Default / Unlimited Tolerance): The batch job tolerates unlimited failures. Even if 90% of records fail, Mule continues processing all remaining records through all steps until the dataset is exhausted.
• maxFailedRecords="0" (Zero Tolerance / Fail-Fast): As soon as the first record encounters an unhandled failure, the Batch Job immediately aborts. Remaining records in the queue are discarded, all subsequent steps are skipped, and the job transitions immediately to the <batch:on-complete> phase.
• maxFailedRecords="N" (Positive Integer): The batch job tolerates up to N failed records. As soon as the (N + 1)th record fails, the batch job aborts immediately and routes to <batch:on-complete>.

---

5. The Batch Aggregator Scope (<batch:aggregator>)

In many integration patterns, processing records individually through steps is necessary for validation and enrichment, but inserting them one-by-one into a destination system (such as Salesforce, SAP, or MySQL) creates disastrous network overhead.

The <batch:aggregator> scope resides inside a <batch:step> and buffers individual processed records into arrays of records before executing bulk operations.

+-----------------------------------------------------------------------------------------+
|                           BATCH AGGREGATOR ARCHITECTURE                                 |
|                                                                                         |
|   <batch:step name="Enrich_And_Aggregate">                                              |
|                                                                                         |
|   [Individual Record] ---> Step Processors (payload = Single Object { id: 101 })        |
|                                     |                                                   |
|                                     v                                                   |
|   +---------------------------------------------------------------------------------+   |
|   | <batch:aggregator size="200">                                                   |   |
|   |                                                                                 |   |
|   |   Buffers incoming records until 200 records accumulate (or step completes)     |   |
|   |                                                                                 |   |
|   |   Inside Aggregator:                                                            |   |
|   |     - payload = Array<Object> (size = 200) [ {id: 101}, {id: 102}, ... ]        |   |
|   |                                                                                 |   |
|   |   Single Bulk Operation:                                                        |   |
|   |     - <salesforce:create-job-bulk-api-v2> or <db:bulk-insert>                   |   |
|   +---------------------------------------------------------------------------------+   |
+-----------------------------------------------------------------------------------------+

Aggregator Types & Configuration:

1. Size-Based Aggregator (size="N"): Accumulates records until size reaches N. Once N records arrive, child message processors execute with payload as an Array of N items. When the batch step finishes, any remaining buffered records (fewer than N) are processed as a final smaller array.

2. Streaming Aggregator (streaming="true"): Streams records sequentially through the aggregator without buffering fixed-size arrays in memory, suitable for writing directly to disk or file streams.

XML Blueprint: High-Performance Salesforce Bulk Upsert

<batch:step name="Step_Salesforce_Upsert" acceptPolicy="NO_FAILURES">
    <!-- 1. Per-record transformation -->
    <ee:transform doc:name="Map to SFDC Account Format">
        <ee:message>
            <ee:set-payload><![CDATA[%dw 2.0
output application/java
---
{
    AccountNumber: payload.erpId,
    Name: payload.customerName,
    AnnualRevenue: payload.revenue as Number default 0,
    BillingCity: payload.city
}]]></ee:set-payload>
        </ee:message>
    </ee:transform>
    
    <!-- 2. Bulk Aggregator: Accumulate 200 records for Bulk Upsert -->
    <batch:aggregator size="200" doc:name="Aggregate 200 Records">
        <!-- Inside aggregator: payload is Array<Object> of 200 accounts -->
        <logger level="INFO" message="#['Pushing bulk batch of ' ++ sizeOf(payload) ++ ' records to Salesforce']" />
        
        <salesforce:upsert 
            config-ref="Salesforce_Config" 
            objectType="Account" 
            externalIdFieldName="AccountNumber" 
            doc:name="Bulk Upsert 200 Accounts">
            <salesforce:records><![CDATA[#[payload]]]></salesforce:records>
        </salesforce:upsert>
    </batch:aggregator>
</batch:step>

[!IMPORTANT] Payload Structure Inside vs. Outside Aggregator

• Outside <batch:aggregator> (inside the step): payload is a single record object (e.g., { id: 101, name: "Acme" }).
• Inside <batch:aggregator>: payload is an Array of records (e.g., [ { id: 101 }, { id: 102 }, ... ]).

---

6. Granular Error Isolation with Try Scopes inside Batch Steps

If you want to handle errors within a batch step without causing the record to be flagged as FAILED, wrap the risky operation inside a <try> scope with <on-error-continue>:

<batch:step name="Resilient_Lookup_Step">
    <try doc:name="Try Lookup">
        <http:request method="GET" config-ref="Tax_API_Config" path="/rates/{country}">
            <http:uri-params><![CDATA[#[{'country': payload.country}]]]></http:uri-params>
            <http:target>vars.taxRate</http:target>
        </http:request>
        <error-handler>
            <on-error-continue doc:name="Fallback on Error">
                <!-- Set default rate so record remains SUCCESSFUL -->
                <set-variable variableName="taxRate" value="#[0.05]" doc:name="Default Tax Rate" />
            </on-error-continue>
        </error-handler>
    </try>
</batch:step>

Because <on-error-continue> handles the exception cleanly within the step, the record is not marked as failed, and it continues smoothly to subsequent NO_FAILURES batch steps.

---

7. Exam Watch: Core Batch Step & Aggregator Scenarios

[!IMPORTANT] acceptPolicy Defaults to NO_FAILURES If acceptPolicy is omitted from <batch:step>, it defaults to NO_FAILURES. Any record that fails in Step 1 will automatically bypass Step 2 unless Step 2 explicitly specifies acceptPolicy="ONLY_FAILURES" or acceptPolicy="ALL".

[!WARNING] acceptExpression vs. acceptPolicy Evaluation Order Mule evaluates acceptPolicy first. If the record's failure state does not match the acceptPolicy, acceptExpression is not even evaluated and the record skips the step immediately.

[!TIP] maxFailedRecords="-1" vs maxFailedRecords="0" Remember: -1 = tolerate all failures (process full dataset). 0 = stop on first failure. Any positive number N = stop when failure count reaches N + 1.