2.1 Custom Oak Index Definitions & Lucene Query Optimization

2.1 Custom Oak Index Definitions & Lucene Query Optimization

Core Principle: In Adobe Experience Manager (AEM), repository queries never execute by scanning content trees directly unless indexing fails completely. The Apache Jackrabbit Oak query engine relies on cost estimation across specialized indexes registered under /oak:index. Mastering Oak Lucene index definitions, property rules, and diagnostic tools is essential to maintaining high-performance Sites applications and preventing catastrophic repository traversal.

---

1. Apache Jackrabbit Oak Architecture & Query Engine

To understand query execution in AEM, one must first understand the architectural decoupling between storage and querying in Apache Jackrabbit Oak.

+------------------------------------------------------------------+
|                   Sling / Application Layer                      |
|             (ResourceResolver, QueryManager, JCR API)            |
+------------------------------------------------------------------+
                                 |
                                 v
+------------------------------------------------------------------+
|                     Oak Query Engine                             |
|     [Query Parser: JCR-SQL2 / XPath] -> [Query AST]              |
|                           |                                      |
|                           v                                      |
|               Cost Calculation & Index Selection                 |
+------------------------------------------------------------------+
             /                   |                   \
            v                    v                    v
   +------------------+ +------------------+ +------------------+
   |  Property Index  | | Oak Lucene Index | |  TraverseIndex   |
   |  (/oak:index/*)  | |  (/oak:index/*)  | |    (Fallback)    |
   +------------------+ +------------------+ +------------------+
            \                    |                   /
             +-------------------+------------------+
                                 |
                                 v
+------------------------------------------------------------------+
|                      Oak NodeStore Layer                         |
|         SegmentNodeStore (TarMK)  |  DocumentNodeStore (Mongo/RDB)|
+------------------------------------------------------------------+

The NodeStore Separation

Oak persists content trees as immutable node states within a NodeStore. AEM 6.5 may use SegmentNodeStore or DocumentNodeStore depending on topology; AEM as a Cloud Service uses Composite NodeStore with release-specific immutable storage and managed mutable DocumentNodeStore-backed content. The NodeStore is an append-only tree optimized for fast read commits, transactional isolation, and revision tracking. However, it possesses no native capability to evaluate relational filters, regex patterns, or full-text searches across millions of paths.

The Query Engine Pipeline

When an application initiates a query via the JCR API, the query undergoes four distinct processing phases:

1. Parsing: Oak parses the incoming query string into an Abstract Syntax Tree (AST).
2. Index Cost Evaluation: The query engine queries every registered QueryIndex service to determine if it can satisfy the AST's constraints.
3. Plan Selection: Oak calculates an estimated numeric cost for each candidate index and selects the index with the lowest cost.
4. Execution & Access Control: The selected index streams matching node paths to Oak. Oak verifies read permissions (AccessControlManager) against the requesting JCR session before returning matching nodes to the caller.

The Danger of Traversal Fallback

If no index can satisfy the query constraints, Oak falls back to the TraverseIndex. Traversal sequentially reads every child node in the repository beneath the query's path restriction:

• Traversal warning: With common defaults, a query that reads about 10,000 nodes logs a traversal warning. The configured limit can differ: *WARN* ... org.apache.jackrabbit.oak.query.QueryImpl Traversal query read 10000 nodes...
• Read-limit abort: With a common 100,000-read limit, Oak aborts a query that crosses the configured threshold and throws: org.apache.jackrabbit.oak.query.RuntimeNodeTraversalException: The query read or traversed more than 100000 nodes

In a production environment, traversal saturates disk I/O, floods the JVM heap with transient node states, triggers high CPU utilization, and risks causing an out-of-memory (OOM) outage.

---

2. JCR Query Languages & Cost Evaluation Mechanics

JCR-SQL2 vs. XPath

In modern AEM development, queries are authored in either JCR-SQL2 or XPath:

AD0-E128 Exam Note: Oak internally translates all XPath queries into JCR-SQL2 representations before cost evaluation. Performance is identical; however, JCR-SQL2 is the tested standard for enterprise index definitions.

Cost Calculation Algorithm

Every index implementation under /oak:index exposes a getCost(Filter filter, NodeState root) method. When evaluating candidate indexes for a given query filter, the index estimates the number of node reads required:

$\text{Cost} = \text{Estimated Matching Nodes} + \text{Path Evaluation Overhead} + \text{Property Comparison Cost}$

Consider this JCR-SQL2 query:

SELECT [jcr:path] FROM [cq:Page] AS page
WHERE ISDESCENDANTNODE(page, '/content/wknd/us/en')
  AND page.[jcr:content/cq:tags] = 'wknd:activity/cycling'
  AND page.[jcr:content/cq:lastModified] > CAST('2026-01-01T00:00:00.000Z' AS DATE)

Oak evaluates candidate indexes as follows:

• nodetype index: Returns a cost proportional to all cq:Page nodes in the entire repository (e.g., 50,000).
• cqPageLucene index: Evaluates path restriction (/content/wknd/us/en), property equality on cq:tags, and date range on cq:lastModified. It estimates matching documents at ~120 nodes and reports a cost of ~125.0.
• TraverseIndex: Reports a cost equal to Double.POSITIVE_INFINITY or the total count of repository nodes under /content/wknd/us/en.

Because cqPageLucene reports the lowest cost (125.0 vs 50,000 vs Infinity), Oak selects cqPageLucene as the query execution plan.

Evaluating Query Plans with Explain Query

Developers must diagnose query plans before deploying code to production. Prepending the keyword EXPLAIN to any query outputs the execution plan chosen by Oak:

EXPLAIN SELECT * FROM [cq:Page] AS a 
WHERE ISDESCENDANTNODE(a, '/content/wknd') 
  AND a.[jcr:content/sling:resourceType] = 'wknd/components/page'

Output in query diagnostics:

[cq:Page] as [a] /* oak:index/cqPageLucene-2-custom-1 
  [sling:resourceType=wknd/components/page] 
  where ([a].[jcr:primaryType] = 'cq:Page') 
  and isdescendantnode([a], [/content/wknd]) */

Diagnostic tools available to developers:

1. AEM Web Console Explain Query Tool: Located at /libs/granite/operations/content/diagnosis/tool.html/que (or Tools > Operations > Diagnosis > Query Performance). Provides query time, execution plans, and slow query logs.
2. Developer Console (AEM as a Cloud Service): Provides an "Explain Query" tab under the environment status dashboard to test queries against active Cloud Service author and publish tiers.

---

3. Oak Index Types: Property Index vs. Oak Lucene

There are two primary index types utilized in AEM Sites development:

Design Warning: Choose index type from query shape and data cardinality. A property index can become large and costly when it tracks highly variable or frequently updated values; analyze the query and prefer an appropriate Lucene definition when range, full-text, ordering, or multi-property behavior is required. Large indexes are a performance and maintenance concern, not automatic repository corruption.

---

4. Anatomy of an Oak Lucene Index Definition

All custom and out-of-the-box indexes reside under /oak:index. Below is a production-grade Lucene index definition for a custom page catalog under /oak:index/cqPageLucene-2-custom-1:

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:jcr="http://www.jcp.org/jcr/1.0"
          xmlns:nt="http://www.jcp.org/jcr/nt/1.0"
          xmlns:oak="http://jackrabbit.apache.org/oak/ns/1.0"
    jcr:primaryType="oak:QueryIndexDefinition"
    type="lucene"
    compatVersion="{Long}2"
    async="[async,nrt]"
    evaluatePathRestrictions="{Boolean}true"
    reindex="{Boolean}false">
    <indexRules jcr:primaryType="nt:unstructured">
        <cq:Page jcr:primaryType="nt:unstructured">
            <properties jcr:primaryType="nt:unstructured">
                <cqTags
                    jcr:primaryType="nt:unstructured"
                    name="jcr:content/cq:tags"
                    propertyIndex="{Boolean}true"
                    facets="{Boolean}true"/>
                <pageTitle
                    jcr:primaryType="nt:unstructured"
                    name="jcr:content/jcr:title"
                    analyzed="{Boolean}true"
                    propertyIndex="{Boolean}true"/>
                <eventDate
                    jcr:primaryType="nt:unstructured"
                    name="jcr:content/eventDate"
                    type="Date"
                    ordered="{Boolean}true"
                    propertyIndex="{Boolean}true"/>
                <status
                    jcr:primaryType="nt:unstructured"
                    name="jcr:content/status"
                    propertyIndex="{Boolean}true"
                    nullCheckEnabled="{Boolean}true"/>
            </properties>
        </cq:Page>
    </indexRules>
</jcr:root>

Critical Node and Property Attributes Explained

• type="lucene": Designates Apache Lucene as the underlying indexing engine.
• compatVersion=2: Enables the newer Lucene index compatibility behavior used by many modern definitions; preserve the setting from the supported base definition when extending it.
• evaluatePathRestrictions=true: Critical property. Instructs Lucene to index the ancestry path hierarchy. When queries include ISDESCENDANTNODE('/content/wknd'), Lucene evaluates the path directly inside the index rather than reading and filtering paths in AEM memory.
• async="[async,nrt]": Configures the asynchronous indexing lanes handling this index.
• indexRules/<nodeType>: Defines the target JCR node types indexed. Properties are only indexed if they belong to a node matching this primary type (e.g., cq:Page).

Analyzing Property Definition Flags

Inside indexRules/<nodeType>/properties/<propNode>, individual properties are configured with targeted flags:

---

5. Asynchronous Indexing Lanes & Oak Mechanics

Why does Oak update Lucene indexes asynchronously rather than synchronously during content writes?

Content Author Saves Page 
       |
       v
JCR Commit -> NodeStore Transaction Committed (Instant Synchronous Response)
       |
       v (Repository Checkpoint Created)
Oak Async Indexer Thread (Runs every 5 seconds)
       |
       +--> Compares Checkpoint diffs
       +--> Extracts modified properties
       +--> Builds Lucene Documents
       +--> Commits Lucene Segments to DataStore / Disk

If Lucene index updates were synchronous, every page publish or dialog save would be blocked while Lucene analyzes text, extracts tokens, and writes segment files. This would severely throttle write throughput.

Indexing Lanes

Oak manages several asynchronous indexing queues, termed lanes:

1. async lane: The standard background indexing lane. Runs periodically (default interval: every 5 seconds) to index standard property and full-text updates.
2. nrt (Near Real-Time) lane: Uses Lucene in-memory searchers and RAMDirectories. Provides sub-second query visibility for newly committed nodes on local author instances before changes are flushed to persistent segments.
3. fulltext-async lane: Used in large AEM Assets deployments for heavy binary document extraction (Apache Tika parsing of PDFs, Word docs) to ensure heavy binary extraction does not block lightweight metadata indexing in the async lane.

---

6. Safe Reindexing Procedures & Cloud Service Conventions

The reindex=true Flag & Production Risks

To reindex an existing index, an administrator sets the property reindex = true on the index definition node (/oak:index/<indexName>).

<jcr:root ... reindex="{Boolean}true" />

During the next execution of the async indexing job, Oak detects reindex=true, wipes all existing Lucene index files, and iterates through every node in the repository to rebuild the index from scratch. Once completed, Oak automatically flips reindex back to false.

[!CAUTION] Production Hazard: Setting reindex=true directly on an active production instance is dangerous. While the index is rebuilding, queries relying on that index immediately fall back to TraverseIndex, potentially causing massive CPU spikes, thread starvation, and service downtime.

AEM as a Cloud Service Naming Convention

In AEM as a Cloud Service (AEMaaCS), modifying out-of-the-box indexes in-place or setting reindex=true in Git code is strictly forbidden. Cloud Manager enforces a strict naming convention for custom and modified indexes:

$\text{Pattern: } \mathbf{<indexName>-<version>-custom-<customVersion>}$

• OOTB Base Index: cqPageLucene-2
• First Customization: cqPageLucene-2-custom-1
• Second Customization: cqPageLucene-2-custom-2
• New Custom Index: wkndProducts-1-custom-1

Blue/Green Index Deployment in Cloud Service

Cloud Manager builds and deploys indexes using a zero-downtime Blue/Green indexing lifecycle:

1. During pipeline execution, Cloud Manager deploys the new index definition (cqPageLucene-2-custom-1).
2. An ephemeral indexing instance runs asynchronously in the background against a read-only snapshot of the repository.
3. Production traffic continues utilizing the old index (cqPageLucene-2). No traversal occurs.
4. Once the new index reaches 100% synchronization and passes health checks, query traffic is atomically switched to cqPageLucene-2-custom-1.
5. The old index is cleanly retired and garbage collected.

---

7. Index Deployment via ui.apps & FileVault Rules

Index definitions must be deployed as part of the immutable application codebase within the ui.apps Maven module.

Repository Location

In the project source tree: ui.apps/src/main/content/jcr_root/_oak_index/cqPageLucene-2-custom-1/.content.xml

FileVault Packaging Rules (filter.xml)

In ui.apps/src/main/content/META-INF/vault/filter.xml, you must define an explicit filter rule with the correct import mode:

<?xml version="1.0" encoding="UTF-8"?>
<workspaceFilter version="1.0">
    <!-- CORRECT: Merges custom index definition without touching sibling indexes -->
    <filter root="/oak:index/cqPageLucene-2-custom-1" mode="merge"/>
</workspaceFilter>

---

8. Common Exam Traps & Troubleshooting Summary

• Trap 1: Sorting without ordered=true: Authoring queries with ORDER BY [jcr:content/date] when ordered=true is omitted causes Oak to buffer the entire result set in memory, triggering OakOutOfMemoryException.
• Trap 2: Missing evaluatePathRestrictions=true: If queries use ISDESCENDANTNODE() but evaluatePathRestrictions is false, Lucene returns matches across the entire repository, and Oak must inspect every path manually in memory.
• Trap 3: Deploying index changes via CRXDE Lite: In AEMaaCS, /oak:index is part of the immutable repository at runtime. Any manual changes via CRXDE Lite or JMX are blocked or lost on the next pod restart.