11.2 Centralized Observability with Amazon OpenSearch & Athena

Centralized Observability: Operational Search vs. Historical Data Lake

Enterprise DevOps architectures generate massive volumes of heterogeneous log telemetry: application stdout logs, ALB access logs, VPC Flow Logs, AWS WAF inspection logs, and AWS CloudTrail audit events. A central responsibility of the DevOps Engineer Professional is architecting a cost-effective, high-performance telemetry pipeline that satisfies two distinct operational requirements:

1. Real-Time Operational Investigation: Near-real-time ingestion, low-latency search, live tailing, anomaly detection, and operational dashboards for incident triage and active debugging.
2. Historical Ad-Hoc Analytics & Compliance: Cost-effective retention of terabytes to petabytes of structured log data queried periodically for security retrospectives, audit compliance, and capacity planning.

---

Amazon OpenSearch Service Architecture & Ingestion

Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) is a managed distributed search and analytics suite. In enterprise deployments, OpenSearch clusters are architected into distinct node tiers to ensure cluster stability under heavy indexing workloads.

Cluster Topologies and Shard Optimization

• Dedicated cluster manager nodes (legacy/API term: dedicated master nodes): Perform cluster coordination, index creation, shard routing, and cluster-state management without serving data queries. For production domains that use dedicated cluster managers, three nodes across three Availability Zones is the standard quorum topology.
• Data Nodes: Store indexed shards and process search and indexing requests. Configured across multiple AZs (Multi-AZ with Standby provides a 99.99% SLA with automatic failover).
• Storage Tiers & Cost Optimization:
  • Hot Tier: Utilizes high-performance local NVMe SSDs or Amazon EBS gp3 volumes for active write and high-frequency read operations.
  • UltraWarm Tier: Backed by Amazon S3 with warm compute caches. Reduces storage costs by up to 90% compared to hot EBS storage while retaining read-only search capabilities.
  • Cold Storage Tier: Fully decouples compute from storage; data resides purely on S3. When historical indices are needed, they are attached to compute on-demand.
• Index State Management (ISM): Automates lifecycle transitions. For example, an ISM policy can automatically roll over an active index when it reaches 50 GB or 1 day, migrate it to UltraWarm after 7 days, transition to Cold after 30 days, and permanently delete it after 365 days.

{
  "policy": {
    "description": "Hot-UltraWarm-Delete Lifecycle Policy",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [{"rollover": {"min_index_age": "1d", "min_primary_shard_size": "40gb"}}],
        "transitions": [{"state_name": "warm", "conditions": {"min_index_age": "7d"}}]
      },
      {
        "name": "warm",
        "actions": [{"warm_migration": {}}],
        "transitions": [{"state_name": "delete", "conditions": {"min_index_age": "90d"}}]
      },
      {
        "name": "delete",
        "actions": [{"delete": {}}]
      }
    ]
  }
}

[!IMPORTANT] Shard Sizing Rule of Thumb: Shards that are too small result in excessive Lucene index overhead and JVM memory pressure (the "over-sharding" disaster). Shards that are too large cause slow rebalancing and recovery. Aim for 10 GB to 30 GB per shard for search-heavy workloads and 30 GB to 50 GB per shard for time-series log ingestion.

OpenSearch Serverless & OpenSearch Ingestion (OSI)

For variable or unpredictably scaling workloads, OpenSearch Serverless eliminates cluster provisioning, cluster manager node configuration, and shard sizing:

• Compute is decoupled from storage and metered in OpenSearch Compute Units (OCUs), where 1 OCU provides 6 GiB of RAM and corresponding vCPU.
• Data is stored durably in Amazon S3; compute capacity scales up during traffic surges and scales down automatically during idle periods.
• Collections are specialized by workload type: Timeseries (optimized for logs and metrics), Search (catalog and e-commerce search), or VectorSearch (semantic AI embeddings).

OpenSearch Ingestion (OSI) is a fully managed, serverless log pre-processor based on Data Prepper. It replaces self-hosted Logstash clusters on EC2:

• Receives logs from Amazon Kinesis, CloudWatch Logs, Fluent Bit, or S3.
• Executes inline parsing (Grok filters), timestamp extraction, field mutations, and data masking (PII redaction).
• Natively sinks enriched documents directly into OpenSearch domains or Serverless collections, with automatic Dead-Letter Queue (DLQ) routing to Amazon S3 for unparseable payloads.

---

Streaming Pipelines: CloudWatch Subscriptions & Kinesis Firehose Dynamic Partitioning

To move logs from source environments (EC2, ECS, Lambda, EKS) into OpenSearch and S3, AWS provides native streaming integrations centered on CloudWatch Logs subscription filters and Amazon Kinesis Data Firehose.

Subscription Filters

A CloudWatch Logs Subscription Filter processes matching events as they are ingested and forwards them to a supported destination:

1. Amazon Data Firehose for buffered delivery to a configured destination.
2. Amazon Kinesis Data Streams for fan-out and multi-consumer streaming analytics.
3. AWS Lambda for custom transformation or forwarding.
4. Amazon OpenSearch Service, using the CloudWatch Logs subscription integration and its Lambda forwarding function.

A log group can have multiple subscription filters, so independent delivery paths can feed the hot OpenSearch tier and the S3 archive.

Kinesis Data Firehose with Dynamic Partitioning

Historically, Firehose delivered data to S3 using a flat, time-based prefix: s3://bucket/YYYY/MM/DD/HH/. When logs from multiple microservices were aggregated, analytical queries with Amazon Athena were forced to scan all services simultaneously, resulting in massive query costs.

Dynamic Partitioning solves this by evaluating the contents of each streaming JSON record using JQ expressions or an inline Lambda function to construct granular S3 prefix paths:

Incoming JSON Record: 
{"timestamp": "2026-09-11T14:32:00Z", "service": "payments", "level": "ERROR", "message": "Timeout"}
                                 │
                                 ▼
Firehose JQ Expression Extraction:
.service = "payments"
.timestamp = "2026-09-11"
                                 │
                                 ▼
Dynamic S3 Key Output:
s3://enterprise-logs/service=payments/year=2026/month=09/day=11/payments-2026-09-11-uuid.parquet

• Inline Format Conversion: Firehose can automatically convert incoming JSON records into columnar Apache Parquet or Apache ORC formats before writing to S3, using an AWS Glue Data Catalog table schema.
• Destination Constraint: Dynamic partitioning is supported only when Amazon S3 is the Firehose stream destination, and it must be enabled when the stream is created. A separate delivery path is required to index the same events in OpenSearch.

---

Serverless Log Analytics with Amazon Athena & S3

Amazon Athena allows DevOps engineers to execute standard ANSI SQL queries directly against structured and semi-structured log files in Amazon S3, without provisioning or managing database infrastructure.

Native AWS Log Ingestion Targets

Athena is widely used to analyze AWS service logs written directly to S3:

• ALB Access Logs: High-volume HTTP request logs capturing client IP, latency, backend target status code, user agent, and TLS cipher.
• AWS CloudTrail Logs: API auditing logs recording caller identity, IAM role ARN, source IP, event name, and request parameters.
• Amazon VPC Flow Logs: Network metadata capturing source/destination IP, port, protocol, packet count, and ACCEPT/REJECT status.
• AWS WAF Logs: Deep inspection logs showing rule group evaluations, terminating actions, and inspected web request headers.

Query Performance Optimization Rules

To minimize Athena query costs and optimize query execution time:

1. Use Columnar Formats (Parquet / ORC): Athena charges $5.00 per TB scanned. If an ALB log table has 30 columns and a query selects only client_ip and target_status_code, Parquet allows Athena to scan only those two column stripes, reducing data scanned by up to 85–90% compared to uncompressed raw text.
2. Compress Data: Use Snappy compression with Parquet or Gzip with raw text to minimize S3 I/O transfer volume.
3. Partition by Query Predicates: Partition data by high-cardinality search dimensions (e.g., year, month, day, and service). Queries with WHERE year = '2026' AND month = '09' prune all other partition directories completely.

---

Glue Data Catalog & Partition Projection

When new logs are continuously delivered to S3, Athena must know about newly created partition directories before it can query them. In traditional setups, engineers used one of two methods:

1. MSCK REPAIR TABLE <table_name>: Instructs Athena to scan the entire S3 bucket prefix tree using the S3 ListObjectsV2 API to discover new partition folders.
2. AWS Glue Crawlers: Scheduled batch jobs that crawl S3 paths and register new partitions in the Glue Data Catalog.

The Failure of Traditional Partition Discovery at Scale

In high-volume environments where logs are partitioned by minute or hour across dozens of services, a bucket accumulates hundreds of thousands of S3 prefix directories:

• MSCK REPAIR TABLE scans the S3 prefix tree sequentially. With millions of objects, the command takes several hours, hits S3 API rate limits (503 Slow Down), and frequently exceeds Athena query timeout limits.
• Glue Crawlers incur high operational costs, introduce a 5- to 15-minute metadata lag, and require complex orchestration.

Partition Projection Architecture

Partition Projection completely eliminates MSCK REPAIR TABLE and Glue Crawlers by shifting partition calculation from static storage metadata to in-memory algorithmic generation:

Traditional Query Workflow:
Athena Query ──> Queries Glue Catalog ──> Sequential S3 ListObjects ──> Extreme Latency / Timeouts

Partition Projection Workflow:
Athena Query (WHERE date = '2026/09/11')
      │
      ▼
Reads Table Properties: projection.date.range = 2024/01/01,NOW
      │
      ├─> Algorithmic In-Memory Calculation (Computes S3 URIs instantly)
      ▼
Direct S3 Object Fetch: s3://logs/year=2026/month=09/day=11/
(Zero S3 ListObjects discovery calls, zero crawler costs, immediate execution)

Partition projection is configured directly in the table properties of the AWS Glue Data Catalog:

CREATE EXTERNAL TABLE alb_access_logs (
  type string,
  time string,
  elb string,
  client_ip string,
  target_status_code string,
  request_url string
)
PARTITIONED BY (date string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES ('input.regex' = '([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ...')
LOCATION 's3://my-enterprise-alb-logs-bucket/AWSLogs/111122223333/elasticloadbalancing/us-east-1/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.date.type' = 'date',
  'projection.date.range' = '2024/01/01,NOW',
  'projection.date.format' = 'yyyy/MM/dd',
  'projection.date.interval' = '1',
  'projection.date.interval.unit' = 'DAYS',
  'storage.location.template' = 's3://my-enterprise-alb-logs-bucket/AWSLogs/111122223333/elasticloadbalancing/us-east-1/${date}'
);

When a query executes WHERE date = '2026/09/11', Athena uses the projection rules to construct the exact S3 prefix .../2026/09/11/ instantly, reading the data with zero S3 directory listing overhead.

---

DOP-C02 Exam Watchouts & Troubleshooting