8.2 Core Functions: map, filter, pluck, reduce, groupBy & orderBy

Core Functions: map, filter, pluck, reduce, groupBy & orderBy

Data transformations in enterprise integration rarely involve simple scalar values; they predominantly center around manipulating collections of records—such as arrays of database rows, lists of JSON objects, and key-value maps. DataWeave provides a suite of higher-order functional operations for transforming, filtering, aggregating, and reshaping collections cleanly and declaratively.

---

1. Transforming Arrays with map

The map function iterates over each element of an Array, applies a transformation expression, and returns a new Array containing the transformed elements in corresponding order.

+-----------------------------------------------------------------------------------------+
|                                 ARRAY MAP OPERATION                                     |
|                                                                                         |
|   Input Array: [ { id: 1, name: "A" }, { id: 2, name: "B" } ]                           |
|                                                                                         |
|   Transformation: payload map ((item, index) -> { ... })                                |
|                     |                                                                   |
|                     +---> Item 0: item = { id: 1, name: "A" }, index = 0               |
|                     |     Output 0: { customerId: 1, displayName: "A", seq: 1 }         |
|                     |                                                                   |
|                     +---> Item 1: item = { id: 2, name: "B" }, index = 1               |
|                           Output 1: { customerId: 2, displayName: "B", seq: 2 }         |
|                                                                                         |
|   Output Array: [ { customerId: 1, displayName: "A", seq: 1 },                          |
|                   { customerId: 2, displayName: "B", seq: 2 } ]                         |
+-----------------------------------------------------------------------------------------+

Explicit vs. Shorthand Syntax

DataWeave supports explicit lambda parameter definitions as well as anonymous positional parameters:

Array Transformation Example:

%dw 2.0
output application/json
var rawAccounts = [
    { "acc_no": 1001, "acc_name": "Acme Corp", "active": "Y" },
    { "acc_no": 1002, "acc_name": "Global Dynamics", "active": "N" }
]
---
rawAccounts map ((account, idx) -> {
    accountId: account.acc_no,
    accountName: upper(account.acc_name),
    isActive: account.active == "Y",
    sequenceNumber: idx + 1
})

Behavior with Null Inputs:

If the input expression to map evaluates to null, DataWeave returns null rather than throwing an exception. For example, null map ($) safely evaluates to null.

---

2. Transforming Objects with mapObject

While map operates strictly on Arrays, mapObject iterates over the key-value pairs of an Object and returns a newly transformed Object.

+-----------------------------------------------------------------------------------------+
|                                OBJECT MAPOBJECT OPERATION                               |
|                                                                                         |
|   Input Object: { "firstName": "John", "lastName": "Doe", "age": 30 }                   |
|                                                                                         |
|   Transformation: payload mapObject ((val, key, idx) -> (upper(key)): val)              |
|                     |                                                                   |
|                     +---> Pair 0: val = "John", key = "firstName", idx = 0              |
|                     +---> Pair 1: val = "Doe",  key = "lastName",  idx = 1              |
|                     +---> Pair 2: val = 30,     key = "age",       idx = 2              |
|                                                                                         |
|   Output Object: { "FIRSTNAME": "John", "LASTNAME": "Doe", "AGE": 30 }                  |
+-----------------------------------------------------------------------------------------+

Anonymous Parameter Identifiers for mapObject:

• $: The field's Value.
• $$: The field's Key (as a Key type).
• $$$: The field's zero-based Index (Number).

The Dynamic Key Parentheses Rule

When constructing key-value pairs inside an object in DataWeave, if the key name is dynamic or generated from an expression/variable, it must be enclosed in parentheses (...).

%dw 2.0
output application/json
var user = {
    "first_name": "Alice",
    "last_name": "Smith",
    "email_address": "alice@example.com"
}
---
user mapObject ((value, key, index) -> {
    // Dynamic key must use parentheses (key): value
    (upper(key)): value
})

---

3. Filtering Collections: filter & filterObject

Filtering removes elements or key-value pairs that do not satisfy a specified boolean condition.

filter on Arrays

filter takes an array and a boolean predicate expression. It returns a new array containing only elements where the predicate evaluates to true:

%dw 2.0
output application/json
var products = [
    { "id": 1, "name": "Chair", "price": 45, "inStock": true },
    { "id": 2, "name": "Desk", "price": 250, "inStock": false },
    { "id": 3, "name": "Monitor", "price": 300, "inStock": true }
]
---
// Shorthand filter using $ (current item)
products filter ($.inStock and $.price >= 50)

Output:

[
  {
    "id": 3,
    "name": "Monitor",
    "price": 300,
    "inStock": true
  }
]

filterObject on Objects

filterObject iterates over key-value pairs in an object and retains only those pairs that meet the criteria:

%dw 2.0
output application/json
var rawCustomer = {
    "name": "Global Tech",
    "taxId": null,
    "email": "",
    "phone": "+1-555-0199",
    "notes": null
}
---
// Strip null and empty string properties
rawCustomer filterObject ((value, key) -> value != null and value != "")

Output:

{
  "name": "Global Tech",
  "phone": "+1-555-0199"
}

---

4. Converting Objects to Arrays with pluck

DataWeave does not allow running map directly on an Object. When an integration receives a key-value map and needs to transform it into a JSON array, the pluck function is used.

+-----------------------------------------------------------------------------------------+
|                                 OBJECT PLUCK OPERATION                                  |
|                                                                                         |
|   Input Object:                                                                         |
|   {                                                                                     |
|       "USD": 1.00,                                                                      |
|       "EUR": 0.92,                                                                      |
|       "GBP": 0.78                                                                       |
|   }                                                                                     |
|                                                                                         |
|   Transformation: payload pluck ((val, key, idx) -> { currency: key, rate: val })       |
|                                                                                         |
|   Output Array:                                                                         |
|   [                                                                                     |
|       { "currency": "USD", "rate": 1.00 },                                              |
|       { "currency": "EUR", "rate": 0.92 },                                              |
|       { "currency": "GBP", "rate": 0.78 }                                               |
|   ]                                                                                     |
+-----------------------------------------------------------------------------------------+

Anonymous Parameter Identifiers for pluck:

• $: Value of the current key-value pair.
• $$: Key of the current key-value pair (as a Key type; cast to String using $$ as String).
• $$$: Zero-based index of the entry.

Practical pluck Example:

%dw 2.0
output application/json
var errorDictionary = {
    "ERR_01": "Invalid credentials",
    "ERR_02": "Account locked",
    "ERR_03": "Session expired"
}
---
errorDictionary pluck ((description, code, index) -> {
    code: code as String,
    message: description,
    priority: index + 1
})

---

5. Aggregations & Grouping: reduce, groupBy, orderBy & distinctBy

reduce: Folding Collections into a Single Result

The reduce function iterates over an array and collapses it into a single cumulative output value.

%dw 2.0
output application/json
var orderItems = [
    { "sku": "A1", "price": 25.0, "qty": 2 },
    { "sku": "B2", "price": 50.0, "qty": 1 },
    { "sku": "C3", "price": 10.0, "qty": 4 }
]
---
{
    // Explicit reduce with accumulator default
    totalCost: orderItems reduce ((item, accumulator = 0) -> 
        accumulator + (item.price * item.qty)
    ),
    
    // Shorthand reduce over numbers (Note: $$ is accumulator, $ is item)
    simpleSum: [10, 20, 30, 40] reduce ($$ + $)
}

groupBy: Partitioning Arrays into Maps

groupBy groups elements of an array according to a key-generating expression, returning an Object where each key maps to an Array of matching records:

%dw 2.0
output application/json
var employees = [
    { "name": "Alice", "dept": "Engineering", "salary": 95000 },
    { "name": "Bob", "dept": "Sales", "salary": 75000 },
    { "name": "Charlie", "dept": "Engineering", "salary": 110000 }
]
---
employees groupBy ($.dept)

groupBy Output:

{
  "Engineering": [
    { "name": "Alice", "dept": "Engineering", "salary": 95000 },
    { "name": "Charlie", "dept": "Engineering", "salary": 110000 }
  ],
  "Sales": [
    { "name": "Bob", "dept": "Sales", "salary": 75000 }
  ]
}

orderBy and distinctBy

• orderBy: Sorts an array based on an extraction expression or list of criteria: payload orderBy ($.price) or payload orderBy [$.dept, -$.salary] (prefix - sorts descending).
• distinctBy: Deduplicates an array by retaining only the first item that produces a unique value for the criteria: payload distinctBy ($.email).

Collection Utility Matrix

---

6. Exam Watch: Core Collection Scenarios