8.1 System Log Analysis & Diagnostic Tooling

8.1 System Log Analysis & Diagnostic Tooling

Core Principle: Troubleshooting enterprise Adobe Experience Manager (AEM) instances requires mastering the diagnostic log topology. Because AEM integrates Apache Sling, Apache Felix (OSGi), Apache Jackrabbit Oak (JCR), and the Apache HTTP Server Dispatcher module, issues rarely present in a single log file. Isolating production failures demands understanding the distinct responsibilities of each log file, correlating transactions across logs using unique request identifiers, and performing forensic analysis on Java stack traces.

---

1. AEM Logging Architecture & Log Topology

AEM logging is powered by Apache Sling Commons Log, which embeds Logback Classic and bridges the SLF4J (Simple Logging Facade for Java) API. All OSGi bundles, Sling components, and custom services write log events through SLF4J loggers. The underlying Sling LogManager dynamically configures Logback appenders and log levels at runtime via OSGi configuration management, without requiring JVM restarts.

In standard on-premise and local SDK installations, all primary operational log files reside within the filesystem under the crx-quickstart/logs/ directory:

crx-quickstart/
└── logs/
    ├── error.log        <-- Core application exceptions, OSGi events, WARN/ERROR entries
    ├── request.log      <-- Sling Engine request timing (inbound -> and outbound <-)
    ├── access.log       <-- NCSA combined web traffic access records
    ├── stdout.log       <-- JVM process output, thread dumps, garbage collection
    ├── stderr.log       <-- JVM fatal errors, unhandled process exceptions
    └── history.log      <-- Package Manager, deployment, and repository modification history

Standard Logging Levels

Sling Commons Log supports standard Logback logging levels in increasing order of severity:

$\mathbf{TRACE} < \mathbf{DEBUG} < \mathbf{INFO} < \mathbf{WARN} < \mathbf{ERROR}$

---

2. Anatomy of the 5 Core AEM Log Files

1. error.log (Application Exceptions & OSGi Diagnostics)

The error.log file is the primary diagnostic log for AEM backend engineers. It records all system-level and application-level log statements at or above the configured root threshold (default is INFO), as well as full Java exception stack traces.

Standard Logback pattern format:

23.09.2026 14:15:32.418 *ERROR* [0:0:0:0:0:0:0:1 [1695478532400] GET /content/wknd/us/en/adventure.html HTTP/1.1] com.wknd.core.models.impl.AdventureModelImpl Failed to adapt adventure resource
java.lang.NullPointerException: null
	at com.wknd.core.models.impl.AdventureModelImpl.init(AdventureModelImpl.java:54) [com.wknd.core:1.0.0.SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

Key components of each error.log header line:

• 23.09.2026 14:15:32.418: Precise millisecond timestamp.
• *ERROR*: Severity level framed by asterisks.
• [0:0:0:0:0:0:0:1 [1695478532400] GET ...]: Thread context containing client IP, request start timestamp epoch, HTTP method, request URI, and HTTP protocol version.
• com.wknd.core.models.impl.AdventureModelImpl: Logger category (Java class emitting the log).
• Failed to adapt adventure resource: Explicit log message followed by the root exception and stack trace.

2. stdout.log & stderr.log (JVM Process & Fatal Crash Logs)

The stdout.log and stderr.log files capture operating-system-level standard output streams redirected from the running Java Virtual Machine process:

• JVM Bootstrapping: Output generated before the OSGi framework and Sling Commons Log have initialized (e.g. JVM startup arguments, JVM classpath, initial heap allocations).
• Garbage Collection (GC) Logs: When JVM flags such as -Xlog:gc* are configured, GC pause durations, heap generation transitions (Young/Tenured), and full GC sweeps are written here.
• Fatal JVM Crashes: Catastrophic process failures that prevent Logback from flushing to error.log, such as native memory exhaustion, JVM core segmentation faults (SIGSEGV), or thread dumps initiated via OS signals (kill -3 <pid>).
• Uncaught OutOfMemoryError: When the JVM heap is completely exhausted (java.lang.OutOfMemoryError: Java heap space or Metaspace), the JVM writes fatal termination details to stderr.log.

3. request.log (Sling Engine Execution Tracing)

The Sling Engine request logger maintains a dedicated operational record of every HTTP transaction processed by the AEM instance. Each incoming HTTP request produces exactly two complementary lines:

23/Sep/2026:14:15:32 +0000 [1042] -> GET /content/wknd/us/en/adventure.html HTTP/1.1
23/Sep/2026:14:15:32 +0000 [1042] <- 200 text/html;charset=UTF-8 84ms

• Inbound Record (->): Written the instant the request arrives at the Sling Engine. Contains the timestamp, unique integer Request ID ([1042]), HTTP method, URL, and protocol.
• Outbound Record (<-): Written the instant response execution completes. Contains the identical Request ID ([1042]), HTTP response status code (200), response Content-Type header (text/html;charset=UTF-8), and total execution duration in milliseconds (84ms).

Diagnostic Power of request.log: If request.log contains an inbound record (-> [1042]) with no corresponding outbound record (<- [1042]), the request thread is either currently executing, deadlocked waiting on a thread lock, or terminated by an abrupt JVM process crash.

4. access.log (Web Traffic & Client Records)

Formatted according to the industry-standard NCSA Combined Log Format, access.log records external client traffic reaching the AEM servlet engine:

192.168.1.105 - anonymous [23/Sep/2026:14:15:32 +0000] "GET /content/wknd/us/en/adventure.html HTTP/1.1" 200 45281 "https://wknd.site/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"

Fields include: Client IP address, remote ident, authenticated user identity (anonymous or authenticated JCR username), timestamp, HTTP request line, HTTP status code, total bytes transferred, Referer header, and browser User-Agent string. Use access.log to analyze incoming traffic spikes, identify external scraper bots, and correlate client IP addresses with slow backend requests.

5. dispatcher.log (Web Server & Cache Diagnostics)

Managed by the Dispatcher module (mod_dispatcher.so) inside the Apache HTTP Server web server tier, dispatcher.log records caching decisions, filter rules, and backend renderer communication:

[Tue Sep 23 14:15:32 2026] [D] [pid 4210] Checking filter rules for [/content/wknd/us/en/adventure.html] ...
[Tue Sep 23 14:15:32 2026] [D] [pid 4210] Filter rule [12] matches -> ALLOW
[Tue Sep 23 14:15:32 2026] [D] [pid 4210] Found cached file [/mnt/var/www/html/content/wknd/us/en/adventure.html]
[Tue Sep 23 14:15:32 2026] [I] [pid 4210] "GET /content/wknd/us/en/adventure.html" - cache HIT [0ms]

Common dispatcher.log entries include:

• Filter Evaluations: Checking filter rules for [...] -> matches rule ... -> ALLOW / DENY.
• Cache Decisions: cache HIT, cache MISS, Cache file is older than .stat file (invalidation).
• Activation Flushes: Processing replication agent cache invalidation requests from publish instances (statfile updated).
• Renderer Communication: Connecting to publish renderers, connection timeout drops, and socket retries.

---

3. Stack Trace Forensic Analysis

When diagnosing application failures in error.log, developers must quickly categorize root causes based on exception signatures.

1. NullPointerException (NPE)

Symptom: Component fails to render or servlet returns HTTP 500. Root Cause: In Sling Models or OSGi services, an injected resource, child node, or property value is absent in the repository, and the code failed to guard against null.

// Dangerous pattern: Assumes property always exists
String title = resource.getValueMap().get("title", String.class);
int length = title.length(); // Throws NullPointerException if title property is missing!

Remediation: In Sling Models, annotate optional injections with @Model(defaultInjectionStrategy = DefaultInjectionStrategy.OPTIONAL) or @Optional, specify @Default(values = "..."), and use Optional.ofNullable() or Apache Commons StringUtils.

2. ClassNotFoundException vs. NoClassDefFoundError

These two exceptions frequently appear after deploying custom OSGi bundles, but have distinct root causes:

Remediation: Inspect the bundle manifest in /system/console/bundles. Check Import-Package headers for unsatisfied package ranges. Ensure required third-party libraries are either embedded via bnd-maven-plugin (@bnd:include) or deployed as independent OSGi bundles.

3. PersistenceException & RepositoryException (JCR Write Failures)

Symptom: Custom workflows, ingestion servlets, or scheduled jobs fail during data modification. Root Cause: JCR repository write rejections during resourceResolver.commit() or session.save().

org.apache.sling.api.resource.PersistenceException: Resource at '/content/wknd/data' cannot be modified
    at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit(JcrResourceProvider.java:489)
Caused by: javax.jcr.AccessDeniedException: /content/wknd/data: not allowed to add or modify properties

Diagnostic Checklist:

1. ACL Permissions: Verify the Service User mapped to the executing bundle possesses jcr:modifyProperties, jcr:addChildNodes, or jcr:write on the target repository branch.
2. Concurrent Modification: If multiple threads modify the same node simultaneously, Oak throws InvalidItemStateException or CommitFailedException: OakState0001: Unresolved concurrent change.
3. Node Type Constraints: Attempting to write a property or child node disallowed by the parent node's primary node type or mixin constraints.

---

4. Cross-Log Request Correlation Mechanics

When troubleshooting complex latency or rendering defects, engineers must correlate events across the entire request processing lifecycle.

[1. Client Browser] 
       |
       v (HTTP GET /content/wknd/us/en.html)
[2. Apache / Dispatcher] --------> Logs entry in access_log & dispatcher.log
       |                            (Records cache HIT or MISS, response time)
       v (Forward to Publish:4503)
[3. AEM Jetty / Sling Engine] ---> Logs inbound entry in request.log: -> [1042]
       |                           Logs client IP & query in access.log
       |
       v (Script Resolution & Execution)
[4. OSGi Services & Models] -----> Throws exception; logs stack trace in error.log
       |                            Tagged with thread name or [1042]
       v
[5. Sling Engine Completion] ----> Logs outbound entry in request.log: <- [1042] 500 120ms

Step-by-Step Correlation Procedure

1. Locate the External Transaction: Identify the target request in access.log using the client IP address, endpoint URL, and approximate timestamp.
2. Identify Request ID in request.log: Search request.log for the matching URL and timestamp:

   grep "/content/wknd/us/en.html" crx-quickstart/logs/request.log
   

   Locate the matching inbound entry: 23/Sep/2026:14:15:32 +0000 [1042] -> GET /content/wknd/us/en.html HTTP/1.1 Use the bracketed integer to pair the inbound and outbound lines in request.log; do not assume that exact integer is copied into every error.log format.
3. Inspect Completion Duration: Check the corresponding outbound entry: 23/Sep/2026:14:15:32 +0000 [1042] <- 500 text/html 1250ms Notice the 500 status and excessive 1250ms processing duration.
4. Extract Exceptions from error.log: Search the matching timestamp window, request path, logger, and thread context. If the deployed log pattern includes a shared request value, use it as an additional clue:

   grep -A 30 "14:15:32" crx-quickstart/logs/error.log | grep -B 2 -A 25 "1042"
   

   This isolates the exact Java stack trace generated during the execution of request 1042.
5. Correlate with Dispatcher: Cross-reference with dispatcher.log to determine whether Dispatcher properly forwarded the request or served an obsolete cached variant.

---

5. Cloud Service Log Observability via Adobe I/O CLI

In AEM as a Cloud Service (AEMaaCS), direct SSH connections, local terminal access, and direct /system/console/slinglog modifications are eliminated. Observability is provided via Cloud Manager and the Adobe I/O Extensible CLI (aio).

Available Log Types in Cloud Service

Real-Time Log Streaming with aio

Developers stream live container logs directly to their local development terminal using the aio-cli-plugin-aem-cloud-manager plugin:

# Stream live error logs from the Author tier in production
aio cloudmanager:tail-logs 5678 author aemerror

# Stream live request logs from the Publish tier
aio cloudmanager:tail-logs 5678 publish aemrequest

# Stream Dispatcher HTTP error logs
aio cloudmanager:tail-logs 5678 dispatcher httpderror

Downloading Historical Log Archives

For forensic post-incident reviews or automated log ingestion into SIEM tools (e.g. Splunk, Datadog), download full historical log dumps:

aio cloudmanager:download-logs 5678 publish aemerror 2

---

6. Exam Traps & Diagnostic Best Practices

• Trap: Confusing access.log and request.log: Remember that access.log records standard web client details (IP, User-Agent, bytes transferred), whereas request.log is unique to Apache Sling and records internal execution durations in milliseconds with -> and <- markers.
• Trap: Missing Transitive Dependencies vs. Bad Bundle Code: If a bundle fails to activate with ClassNotFoundException, beginners often suspect syntax bugs in Java code. The AD0-E128 exam tests your knowledge that this is an OSGi package wiring defect; inspect Import-Package in the bundle manifest.
• Trap: Searching error.log for Client Latency: Network round-trip delays between the browser and Dispatcher do not appear in error.log. Backend processing latency inside the JVM is exclusively measured in request.log (duration in ms).