11.1 Log Management on Grail: Storage, Ingestion Pipelines & Log Masking

Enterprise application architectures generate immense volumes of unstructured log telemetry across microservices, serverless runtimes, container orchestrators, and legacy virtual machines. In traditional log monitoring paradigms, analyzing these massive event streams required deploying complex indexing architectures—most notably Elasticsearch or Lucene-based clusters. These legacy systems depend on inverted indices, pre-defined schema mappings, and continuous shard rebalancing.

However, traditional indexed logging architectures suffer from severe operational and financial bottlenecks:

• Index Bloat and Storage Ballooning: Inverted indices frequently consume two to three times more disk storage than the raw log data itself.
• Schema Lock-in and Mapping Explosions: Dynamic application logging often introduces new fields that conflict with existing index mappings, causing ingestion failures or dropped records.
• Costly Re-Indexing: Modifying retention policies, field data types, or parsing rules historically required re-indexing terabytes or petabytes of historical data, consuming massive compute resources.
• Decoupled Telemetry Silos: Logs in legacy systems exist in isolation from infrastructure metrics, distributed traces, and application topology, forcing site reliability engineers (SREs) to manually correlate timestamps during high-severity outages.

Dynatrace eliminates these architectural trade-offs through Dynatrace Grail, an indexless data lakehouse specifically engineered for observability telemetry. Grail unifies logs, metrics, traces, and entity topology into a single causal analytical model powered by Schema-on-Read mechanics and Massive Parallel Processing (MPP).

---

1. Grail Lakehouse Architecture: Indexless Storage & Schema-on-Read

At the core of modern Dynatrace Log Management is Grail, a petabyte-scale, column-oriented analytical store built directly on cloud object storage (such as AWS S3, Azure Blob Storage, or Google Cloud Storage). Unlike relational databases or indexed search engines, Grail does not build or maintain inverted indices upon log ingestion.

+---------------------------------------------------------------------------------------------------+
|                                 GRAIL DATA LAKEHOUSE ARCHITECTURE                                 |
+---------------------------------------------------------------------------------------------------+
| Ingestion Boundary -> Ingest-Time Rules (Masking, Multi-line, Storage Matchers)                   |
|       │                                                                                           |
|       ▼                                                                                           |
| Immutable Cloud Object Storage (Indexless, Compressed, Columnar Micro-Partitions)                 |
|       │                                                                                           |
|       ▼                                                                                           |
| Massively Parallel Processing (MPP) Vectorized Compute Query Nodes                                |
|       │                                                                                           |
|       ▼                                                                                           |
| Schema-on-Read Evaluation: DQL parses, filters, projects, and aggregates at query runtime         |
|       │                                                                                           |
|       ▼                                                                                           |
| Unified Context: Automatic Smartscape Topology Stitching (Host, Process Group, Pod, Service)      |
+---------------------------------------------------------------------------------------------------+

The Schema-on-Read Paradigm

In Grail, logs are ingested and written into compressed, immutable micro-partitions in their raw, native format alongside standard metadata attributes. No index is generated, and no rigid schema is enforced during write operations. Instead, data structure is applied dynamically at read time via the Dynatrace Query Language (DQL).

This architectural design yields critical operational advantages:

1. Zero Index Maintenance: Operations teams never manage shards, re-index data, or troubleshoot index mapping explosions.
2. Infinite Analytical Flexibility: If a developer introduces a new nested JSON attribute or custom logging format, engineers can immediately parse, extract, and aggregate that attribute across historical logs using DQL without retroactive schema migrations.
3. Radical Cost and Storage Efficiency: By avoiding index generation, Grail achieves superior compression ratios and drastically reduces raw storage overhead compared to legacy search indices.

Native Smartscape & Davis AI Integration

Logs in Grail are never orphaned text strings. Whenever OneAgent or an ActiveGate captures a log event, Dynatrace automatically enriches the log record with contextual topology metadata linking it to the Smartscape dependency model:

• dt.entity.host: The unique Dynatrace identifier of the underlying host.
• dt.entity.process_group_instance: The specific running process instance emitting the log.
• dt.entity.service: The logical software service handling the transaction.
• k8s.pod.name, k8s.namespace.name, k8s.container.name: Complete container orchestration context.

Because logs are natively topologically stitched, Davis AI continuously correlates log anomalies (such as sudden surges in ERROR or FATAL log rates) with infrastructure saturation, service response time degradations, and PurePath distributed traces. When Davis raises a Problem card, relevant log patterns are automatically identified as causal or contributory events.

---

2. Log Ingestion Mechanisms & Discovery

Dynatrace provides multiple ingestion pathways to capture log telemetry across physical data centers, public cloud providers, container clusters, and serverless environments.

OneAgent Automatic Log Detection

When OneAgent is installed on a host in Full-Stack or Infrastructure Monitoring mode, its dedicated log monitoring module automatically discovers, inspects, and streams log data without requiring third-party shippers or manual agent configurations.

OneAgent discovers log sources through several automated detection routines:

• Process File Descriptor Tracking: OneAgent monitors active process handles. If a discovered process opens or writes to a file matching standard log file naming conventions (e.g., *.log, catalina.out, error_log), OneAgent automatically registers the file as a log source.
• Standard Directory Inspections: OneAgent routinely monitors standard operating system log directories such as /var/log on Linux and %SystemRoot%\System32\Winevt\Logs on Windows.
• Container Output Interception: In containerized environments (Docker, containerd, CRI-O), OneAgent intercepts standard output (stdout) and standard error (stderr) streams directly from container runtimes or host log mount paths.
• Systemd-Journald Integration: On modern Linux systems, OneAgent natively queries the systemd-journald socket to ingest system service units and kernel logs.
• Windows Event Log: On Windows hosts, OneAgent integrates directly with the Windows Event Log API to stream Application, System, and Security event channels.

Environment ActiveGate Log Ingestion & Routing

In secure enterprise networks, direct outbound Internet connectivity from application hosts to Dynatrace SaaS is strictly restricted. Environment ActiveGates serve as secure intermediaries:

• ActiveGates act as local log proxies, receiving encrypted log batches from OneAgents across internal network zones.
• ActiveGates bundle, compress, and stream log telemetry over outbound port 443 to the SaaS Grail lakehouse.
• ActiveGates can also act as collection endpoints for syslog daemons (rsyslog, syslog-ng) and cloud log forwarders.

Direct API Ingestion (POST /api/v2/logs/ingest)

For environments where OneAgent cannot be deployed—such as serverless functions (AWS Lambda, Azure Functions, Google Cloud Run), edge appliances, proprietary network appliances, or third-party log forwarders (Fluentd, Fluent Bit, Logstash, Vector)—Dynatrace provides the high-throughput Log Ingest v2 API (/api/v2/logs/ingest).

The Log Ingest API accepts JSON arrays containing plain-text or structured log objects. Ingested payloads can include custom attributes and OpenTelemetry semantic conventions, which Grail automatically ingests into the logs table.

---

3. Log Ingestion Rules & Storage Configurations

In large enterprise deployments, unconstrained log collection can result in massive volumes of low-value, noisy telemetry (such as verbose health-probe logs or repetitive debug statements) that consume storage capacity and budget. Dynatrace provides Log Storage Rules and Log Processing Rules to give administrators granular control over what telemetry is stored, transformed, or discarded.

Storage Matchers: Inclusion vs. Exclusion

Log storage rules are evaluated at the ingestion boundary before records are committed to persistent Grail storage. Administrators define matchers using boolean expressions based on log attributes:

+---------------------------------------------------------------------------------------------------+
|                                 LOG STORAGE EVALUATION PIPELINE                                   |
+---------------------------------------------------------------------------------------------------+
| Incoming Log Stream (OneAgent, ActiveGate, Log Ingest API)                                       |
|       │                                                                                           |
|       ▼                                                                                           |
| Matcher Evaluation:                                                                               |
|   - Attribute Matching: log.source, status, content, k8s.namespace.name, dt.entity.host          |
|       ├── MATCH EXCLUSION RULE (e.g., status == "DEBUG" and environment == "staging")            |
|       │     └── ACTION: DROP FROM PERSISTENT STORAGE (No Grail DDU consumed)                     |
|       │                                                                                           |
|       └── MATCH INCLUSION RULE (e.g., status in ("ERROR", "WARN") or k8s.namespace == "prod")   |
|             └── ACTION: STORE IN GRAIL (Default 35-day or Custom Extended Retention)              |
+---------------------------------------------------------------------------------------------------+

1. Storage Inclusion Rules: Explicitly define the criteria for logs that must be persisted in Grail. For example, storing all logs where status in ("ERROR", "WARN", "FATAL") or where k8s.namespace.name == "production".
2. Storage Exclusion Rules: Explicitly drop logs matching specific criteria. For example, discarding all records where content like "*healthcheck*" or where status == "DEBUG" originating from non-production host groups.
3. Stream vs. Storage: Logs excluded from storage do not consume Grail storage licensing. However, administrators can still process them in real time for stream analysis, metric extraction, or event triggering before they are discarded.

Configurable Retention Tiers

By default, logs persisted to Grail are retained for 35 days, which satisfies standard operational baselining, troubleshooting, and post-mortem analysis requirements. However, organizations subject to strict regulatory frameworks (such as financial compliance, HIPAA, or SOC 2) can configure custom retention buckets extending storage for specific log sets up to multiple years.

---

4. Multi-Line Log Detection & Exception Aggregation

Modern runtime environments frequently output multi-line records. The most prevalent example is an application exception stack trace in Java, .NET, Python, or Go:

2026-09-10 14:32:05.112 ERROR [OrderService] Payment processing failed
java.lang.NullPointerException: Cannot invoke "Account.getBalance()" because "userAccount" is null
    at com.dynatrace.bank.PaymentProcessor.charge(PaymentProcessor.java:142)
    at com.dynatrace.bank.OrderService.processOrder(OrderService.java:88)
    at com.dynatrace.bank.OrderController.submit(OrderController.java:45)

The Line-Splitting Problem

If a log collector treats every line break (\n) as a distinct log record, the single logical exception above is split into five separate log records:

• Line 1 is captured as an ERROR log.
• Lines 2 through 5 are captured as unassociated, context-free plain text lines, frequently categorized with default INFO or unknown status.

This line-splitting corrupts error baselining, disrupts Davis AI anomaly detection, and makes querying log data in DQL nearly impossible.

Multi-Line Detection Rules

Dynatrace resolves this through Multi-Line Log Detection Rules executed by OneAgent at the point of capture. A multi-line detection rule defines a Header Pattern using regular expressions that delineates the start of a new log record:

• Header Pattern Definition: A regex matching the initial timestamp or severity format of a primary log line, such as ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.
• Aggregation Logic: When OneAgent reads a line matching the header pattern, it begins a new log record. Any subsequent lines that do not match the header pattern (such as lines starting with whitespace, tabs, or at com...) are automatically appended into the content field of the active log record.
• Delimiter Options: Administrators can configure OneAgent to detect multi-line records based on standard conventions (e.g., lines starting with dates, times, or custom brackets) or specify explicit start/end pattern boundaries.

---

5. Log Masking & Sensitive Data Protection

Enterprise logs frequently contain confidential or legally restricted information, including Personally Identifiable Information (PII), authentication credentials, session tokens, and payment data:

• Primary Account Numbers (credit card PANs)
• Social Security Numbers (SSNs) and National ID numbers
• HTTP Authorization: Bearer <token> headers
• Passwords and secret API keys in query parameters

Failure to sanitize this telemetry violates critical regulatory frameworks including PCI-DSS, GDPR, and HIPAA.

Ingest-Time Masking vs. Query-Time Masking

Understanding the architectural distinction between ingest-time masking and query-time masking is critical for the Dynatrace Certified Associate examination:

Exam Key Point: Compliance mandates such as PCI-DSS strictly forbid the storage of unencrypted cardholder data on persistent media. Therefore, Ingest-Time Masking is mandatory. Query-time masking does not satisfy persistent storage compliance requirements.

Ingest-Time Masking Mechanics

Administrators configure Log Masking Rules in Dynatrace Settings (Settings -> Log Monitoring -> Log processing -> Masking). Each rule defines:

1. Matcher Scope: An optional filter limiting the rule to specific log sources, hosts, process groups, or Kubernetes namespaces.
2. Target Expression (Regex): A regular expression matching the sensitive character pattern to be scrubbed.
3. Masking Action:
   • String Replacement: Replaces the matched regex group with a static string (e.g., [REDACTED], [MASKED_SSN], or XXXX-XXXX-XXXX-1234).
   • Cryptographic SHA-256 Hashing: Replaces the sensitive attribute with its one-way cryptographic SHA-256 hash. Hashing preserves the ability for engineers to correlate distinct user sessions or track transaction flows across microservices without exposing the actual identity or plain-text PII of the user.

---

Log Ingestion Methods Comparison