1.1 AEM Dispatcher Configuration & Caching Fundamentals

1.1 AEM Dispatcher Configuration & Caching Fundamentals

Quick Answer: The AEM Dispatcher (mod_dispatcher.so) serves as an Apache HTTP Server caching module and reverse-proxy security barrier. Requests pass through /virtualhosts matching, /filter sequential evaluation (where the last matching rule wins), and /cache inspection. Cache invalidation operates via .stat files whose directory depth is controlled by /statfileslevel. /gracePeriod can briefly serve stale, auto-invalidated content after the last activation to throttle repeated invalidations, while /ignoreUrlParams lets selected tracking parameters use the normal cache entry.

In enterprise Adobe Experience Manager (AEM) architectures, the Dispatcher is the first line of defense and performance optimization. It resides on the web server layer—typically Apache HTTP Server (HTTPD)—operating via the mod_dispatcher.so shared object module. It simultaneously fulfills two critical roles: an aggressive, filesystem-backed caching engine that shields AEM Publish instances from overwhelming traffic, and an application-layer security firewall that blocks unauthorized access to JCR repositories, administrative consoles, and internal servlets.

---

Dispatcher Architecture & Request Handling Flow

When an HTTP request enters the web tier, Apache HTTPD delegates request processing to the Dispatcher module based on configuration directives in httpd.conf and virtual host configuration files (conf.d/vhosts/*.vhost). The Dispatcher module evaluates the request against its configured farms (defined in files such as dispatcher.any or modularized conf.dispatcher.d/farms/*.farm).

The internal request execution sequence follows a strict multi-stage lifecycle:

1. Virtual Host Matching (/virtualhosts): The Dispatcher inspects the incoming Host HTTP header and selects the farm configuration whose /virtualhosts list matches the incoming domain name or wildcard pattern.
2. Security Filtering (/filter): The request URL, HTTP method, query string, selectors, and extension are evaluated against sequential filter rules. If denied, the Dispatcher terminates processing and returns an HTTP 403 Forbidden or 404 Not Found response.
3. Cache Lookup & Document Root (/docroot): If the request passes the filter rules and uses a cacheable HTTP method (typically GET or HEAD), the Dispatcher checks the local filesystem document root (/docroot) for an existing cached file.
4. Validity Verification (.stat comparison): If a cached file exists, the Dispatcher compares its filesystem modification timestamp against the modification timestamp of the nearest .stat file in the directory hierarchy.
   • If the cached file is newer than the .stat file, the Dispatcher immediately serves the static asset directly from disk without contacting AEM.
   • If the cached file is older than the .stat file, it is considered stale (invalidated).
5. Grace Period Inspection (/gracePeriod): For auto-invalidated content, the Dispatcher can continue serving the stale cached resource for the configured number of seconds after the last activation. This throttles repeated .stat invalidations during an activation burst; it is not a per-resource single-flight lock.
6. Render Delegation (/renders): If the asset is missing, stale, or marked uncacheable by /rules or HTTP headers, the request is forwarded to the designated AEM Publish instances defined in /renders.
7. Response Caching & Delivery: AEM processes the request and streams the response back to the Dispatcher. If response headers (such as Dispatcher: no-cache or Cache-Control: private) do not prohibit caching and the request path matches /cache /rules, the Dispatcher writes the response file to /docroot and serves it to the client.

---

Farm Configuration Structure

A Dispatcher configuration file contains one or more named farm blocks. In modern modular configurations (AEM as a Cloud Service and AEM 6.5 archetype standards), each farm lives in a dedicated .farm file under conf.dispatcher.d/farms/.

/publishfarm {
    /clientheaders {
        "*"
    }
    /virtualhosts {
        "www.mybrand.com"
        "mybrand.com"
        "*.mybrand.com"
    }
    /renders {
        /rend01 {
            /hostname "10.0.10.15"
            /port "4503"
            /timeout "60000"
            /receiveTimeout "600000"
            /ipv4 "1"
        }
    }
    /filter {
        $include "../filters/filters.any"
    }
    /cache {
        /docroot "/var/www/html/mybrand"
        /statfileslevel "2"
        /allowAuthorized "0"
        /gracePeriod "2"
        /ignoreUrlParams {
            /0001 { /glob "*" /type "deny" }
            /0002 { /glob "utm_*" /type "allow" }
            /0003 { /glob "gclid" /type "allow" }
        }
        /rules {
            $include "../cache/rules.any"
        }
        /invalidate {
            /0000 { /glob "*" /type "deny" }
            /0001 { /glob "*.html" /type "allow" }
        }
        /allowedClients {
            /0000 { /glob "*.*.*.*" /type "deny" }
            /0001 { /glob "10.0.10.*" /type "allow" }
        }
    }
}

Core Directives Breakdown

---

Security Filtering Rules (/filter)

The /filter section is the core security mechanism of the Dispatcher. It filters incoming HTTP requests before they can touch either the cache or the AEM Publish render nodes.

The Golden Rule: Sequential Evaluation (Last Match Wins)

A critical technical concept frequently tested on the AD0-E128 exam is the evaluation order of Dispatcher filter rules:

Exam Rule: Dispatcher /filter rules are evaluated sequentially from the first rule (/0000) to the final rule. The last matching rule determines whether the request is allowed or denied.

Unlike traditional firewall rules that terminate on first match, the Dispatcher processes the entire list unless a rule specifically terminates evaluation. Consequently, the universal architectural standard is the deny-all whitelist model:

1. Rule /0000 denies everything (/type "deny" /url ".*").
2. Subsequent rules incrementally allow specific public resources, extensions, and content paths.

Modern Pattern Syntax vs Legacy /glob

Legacy AEM configurations used /glob for filtering, which matched against the entire HTTP request line (e.g., GET /content/mypage.html HTTP/1.1). This approach was error-prone, vulnerable to URL encoding exploits, and lacked granularity. Modern AEM configurations use explicit property matching:

/filter {
    # 1. Deny everything by default
    /0000 { /type "deny" /url ".*" }

    # 2. Allow standard HTTP methods for public content
    /0001 { /type "allow" /method "GET" /url "/content(/.*)?" }
    /0002 { /type "allow" /method "HEAD" /url "/content(/.*)?" }

    # 3. Allow client-side libraries
    /0010 { /type "allow" /method "GET" /url "/etc\.clientlibs/.*" }

    # 4. Deny access to administrative tools, consoles, and JCR exploration
    /0020 { /type "deny" /url "/crx.*" }
    /0021 { /type "deny" /url "/system.*" }
    /0022 { /type "deny" /url "/bin.*" }
    /0023 { /type "deny" /url ".*\.infinity\.json" }
    /0024 { /type "deny" /url ".*\.tidy\.json" }
    /0025 { /type "deny" /url ".*\.(json|xml)" /path "/content(/.*)?" }

    # 5. Allow specific public API endpoints
    /0030 { /type "allow" /method "POST" /url "/bin/myproject/public-contact-form" }
}

Properties available within a filter rule include /method, /url, /path, /query, /extension, /selectors, and /suffix. Using regex in /url or /path allows precise boundary enforcement, such as disallowing dangerous selectors (infinity, tidy) that could trigger denial-of-service memory spikes on Publish instances.

---

Cache Management: /docroot, /rules, and Invalidation

Caching in the Dispatcher is file-based. When the Dispatcher caches /content/mybrand/en/products.html, it writes an identical directory structure under the configured /docroot (e.g., /var/www/html/mybrand/content/mybrand/en/products.html).

The /rules Directive

The /cache /rules section dictates which HTTP requests are eligible for filesystem caching. Like filters, rules are evaluated sequentially, and the last matching rule decides eligibility:

/rules {
    /0000 { /type "deny" /glob "*" }
    /0001 { /type "allow" /glob "*.html" }
    /0002 { /type "allow" /glob "*.css" }
    /0003 { /type "allow" /glob "*.js" }
    /0004 { /type "allow" /glob "*.png" }
    /0005 { /type "allow" /glob "*.svg" }
}

Even if /rules allows an asset, the Dispatcher will not cache a response if:

• The request is not an HTTP GET or HEAD.
• The request contains unignored query parameters.
• A response header explicitly forbids caching (Dispatcher: no-cache, Cache-Control: private, Cache-Control: no-cache, or Set-Cookie).
• The /allowAuthorized flag is set to "0" (the default) and the request carries an Authorization header or login cookie.

---

Stat Files Hierarchy: /statfileslevel

The Dispatcher tracks cache invalidation through empty marker files named .stat. When content is activated from AEM Author, a replication flush agent sends an HTTP CQ-Action: Activate request to the Dispatcher. Instead of scanning and deleting thousands of static files, the Dispatcher updates the modification timestamp of the appropriate .stat file.

The /statfileslevel directive determines the depth at which .stat files are placed relative to /docroot:

/docroot "/var/www/html/mybrand"
/statfileslevel "2"

Stat File Placement Matrix

Consider a document root /var/www/html/mybrand containing content for two independent brands and countries: /content/brand-a/us/en/home.html and /content/brand-b/fr/fr/home.html.

Exam Trap: Setting /statfileslevel too high (e.g., level 5 or 6) creates .stat files deep inside leaf folders. When an author publishes a shared asset or common navigation page, the .stat file touched at the leaf level fails to invalidate dependent sibling pages or parent templates. Conversely, setting /statfileslevel "0" on a multi-site installation causes catastrophic cache churn, as updating a single typo on Brand A invalidates millions of cached assets on Brand B.

Throttling Activation Bursts: /gracePeriod

Publishing a batch of related pages can cause repeated auto-invalidations: each activation can touch a .stat file and make matching cached resources stale again. On a busy site, that churn increases the number of cache misses and backend renders while the batch is still arriving.

The Problem Without a Grace Period

1. An author publishes the first item in a batch, and the flush request updates the relevant .stat timestamp.
2. A request causes the stale cached resource to be fetched again from Publish.
3. Another activation in the same batch touches the .stat file again, invalidating the newly cached response.
4. Repeating this cycle increases backend work until the activation batch finishes.

The /gracePeriod Behavior

Setting /gracePeriod (measured in seconds) provides a temporary buffer:

/cache {
    /docroot "/var/www/html/mybrand"
    /statfileslevel "2"
    /gracePeriod "2"
}

The /gracePeriod value defines how long a stale, auto-invalidated resource may still be served after the last activation. This throttles repeated .stat touching during a batch of activations and can reduce the surge of re-fetches. It is not a per-resource lock that guarantees exactly one render request while every concurrent request receives stale content.

---

Marketing Parameters & /ignoreUrlParams

By default, the Dispatcher considers any request with query parameters (e.g., ?utm_source=email) as a dynamic request that must bypass the cache and hit Publish. In modern digital marketing, campaign URLs regularly contain tracking tokens (utm_source, utm_medium, utm_campaign, gclid, fbclid). If unmanaged, every unique tracking string bypasses the cache, dropping cache hit ratios from 95%+ down to near zero.

The /ignoreUrlParams configuration defines which query parameters should be ignored by the caching engine. When ignored parameters are present, the Dispatcher strips them from its cache key lookup and serves the base cached page (products.html), while preserving the parameters in the client's browser for analytics scripts.

/ignoreUrlParams {
    # 1. Deny all query parameters by default (must bypass cache)
    /0001 { /glob "*" /type "deny" }
    
    # 2. Whitelist marketing tracking parameters
    /0002 { /glob "utm_*" /type "allow" }
    /0003 { /glob "gclid" /type "allow" }
    /0004 { /glob "mc_cid" /type "allow" }
    /0005 { /glob "mc_eid" /type "allow" }
}

Exam Trap: Do not confuse /ignoreUrlParams with Apache mod_rewrite query string manipulation. mod_rewrite physically strips or redirects the query string away from the browser, which destroys client-side Google Analytics or Adobe Analytics tracking. /ignoreUrlParams leaves the URL intact for the client but instructs the Dispatcher cache engine to disregard the parameters when reading from or writing to /docroot.

---

Flush Security: /allowedClients

Cache invalidation requests are HTTP requests containing custom headers: CQ-Action: Activate or CQ-Action: Delete, accompanied by CQ-Handle: /content/mybrand/en/page. If the flush endpoint is left open to the public internet, malicious actors could forge invalidation headers and continuously wipe the Dispatcher cache, mounting a devastating denial-of-service attack.

The /allowedClients block within /cache secures the invalidation handler by validating the client IP address of the incoming flush request:

/allowedClients {
    # Deny all IP addresses by default
    /0000 { /glob "*.*.*.*" /type "deny" }
    
    # Allow only specific AEM Publish server IP addresses or subnets
    /0001 { /glob "10.0.10.15" /type "allow" }
    /0002 { /glob "10.0.10.16" /type "allow" }
}

In AEM as a Cloud Service, Dispatcher flush is managed automatically by the internal service mesh infrastructure; however, in AEM 6.5, on-premises, and Adobe Managed Services (AMS), restricting /allowedClients to trusted invalidation sources is an important security control.