12.1 Agent vs Gateway Deployment Topologies

12.1 Agent vs Gateway Deployment Topologies

Quick Answer: The OpenTelemetry Collector supports three primary deployment patterns: the Agent pattern (deployed as a Kubernetes DaemonSet, pod sidecar, or host systemd daemon for low-latency local offload, host metrics collection, and local container log scraping), the Gateway pattern (a centralized, horizontally scalable Kubernetes Deployment behind a load balancer that isolates backend credentials, enforces egress security, and runs compute-heavy OTTL transformations and tail-based sampling), and the Hybrid Two-Tier pattern (the production standard, pairing Tier 1 Node Agents for local enrichment with Tier 2 Central Gateways for sampling, PII redaction, and multi-backend routing).

In enterprise software architectures, telemetry collection is an operational side-channel: it must capture exhaustive runtime data across traces, metrics, and logs without degrading application performance, consuming excessive compute, or exposing sensitive credentials. While language-level OpenTelemetry SDKs can theoretically transmit telemetry directly from microservices to remote software-as-a-service (SaaS) or on-premises backends, this direct-export approach couples application runtimes to specific backend endpoints, requires distributing secret API tokens to hundreds of application containers, and burns application CPU on compression, retries, and network socket management.

Deploying an OpenTelemetry Collector decouples instrumentation from telemetry transport. However, designing where and how the Collector runs is one of the most critical architectural decisions in any observability rollout. This section analyzes the spectrum of Collector deployment patterns, comparing their operational characteristics, security perimeters, and resource profiles.

---

The Spectrum of Collector Deployment Patterns

Collector deployment models exist along a continuum between extreme localization (running right beside the application code) and extreme centralization (running as a shared platform infrastructure service):

+-------------------------------------------------------------------------+
|               Collector Deployment Architecture Spectrum                |
+-------------------------------------------------------------------------+
  Local Agent (Sidecar)  -->  Local Agent (DaemonSet)  -->  Central Gateway
  [Per-Pod Isolation]         [Per-Node Aggregation]        [Cluster-Wide Scale]
  - Zero cross-app impact     - Hostmetrics & logs          - Central secrets
  - High memory footprint     - Low overhead per node       - Tail-based sampling
  - Simple serverless fit     - Local k8sattributes         - Single egress point

In modern cloud-native systems, these patterns resolve into three primary production models:

1. The Agent Pattern: Deployed locally on the compute node or pod alongside the workload.
2. The Gateway Pattern: Deployed as a centralized, scalable service tier shared across workloads.
3. The Hybrid Two-Tier Pattern: A layered combination where local agents offload telemetry to centralized gateways.

---

The Agent Pattern (Node DaemonSet & Pod Sidecar)

In the Agent pattern, the Collector process runs in immediate physical or virtual proximity to the instrumented application. In Kubernetes, this is implemented either as a DaemonSet (one Collector pod per Kubernetes worker node) or as a Sidecar container (one Collector container inside each application pod sharing the localhost network namespace). On virtual machines or bare-metal servers, it runs as a local systemd daemon.

+-------------------------------------------------------------------+
|                    Kubernetes Worker Node                         |
|                                                                   |
|  +----------------------+      +----------------------+           |
|  |  App Pod 1 (OTel)    |      |  App Pod 2 (OTel)    |           |
|  +----------------------+      +----------------------+           |
|             |                             |                       |
|             | OTLP (HostIP:4318)          | OTLP (HostIP:4317)    |
|             v                             v                       |
|  +-------------------------------------------------------------+  |
|  |       Tier 1 Node Agent (Collector DaemonSet Pod)           |  |
|  |                                                             |  |
|  |  - hostmetricsreceiver  <-- mounts /proc and /sys           |  |
|  |  - filelogreceiver      <-- tails /var/log/pods             |  |
|  |  - k8sattributes        <-- resolves client IP to pod meta  |  |
|  +-------------------------------------------------------------+  |
+-------------------------------------------------------------------+
                                  |
                                  v (OTLP over cluster network)
                     [Tier 2 Gateway or Backends]

Operational Responsibilities of the Agent

1. Immediate Telemetry Offload: A sidecar is reached at localhost:4317 (gRPC) or localhost:4318 (HTTP). A DaemonSet agent is reached at the node's IP, which the pod learns through the Kubernetes Downward API (status.hostIP), with the agent exposing a hostPort. Either way the hop is short and local, so SDK export queues (BatchSpanProcessor) stay small.
2. Host Metrics Collection: Because the DaemonSet agent runs with node-level access, it mounts host kernel filesystems (/proc and /sys) to run the hostmetricsreceiver. It collects CPU utilization, memory pressure, disk I/O, network interface counters, and filesystem capacity without needing remote polling credentials.
3. Local Container Log Scraping: The DaemonSet agent mounts /var/log/pods from the host. Using the filelogreceiver, it tails, parses, and checkpoints stdout and stderr log streams generated by all container runtimes on that node.
4. Local Resource Enrichment: Using the k8sattributesprocessor, the agent inspects the source IP address of incoming TCP connections, maps that IP to the local node's pod registry, and automatically enriches every incoming span, metric, and log record with k8s.pod.name, k8s.namespace.name, k8s.node.name, and k8s.pod.uid. When the processor is configured with a node filter (filter: node_from_env_var: KUBE_NODE_NAME), each agent watches only the pods on its own node, which keeps load on the Kubernetes API server low.

Strengths of the Agent Pattern

• Negligible Application Overhead: Applications do not spend CPU cycles on compression algorithms (GZIP, ZSTD), TLS session negotiation, or lengthy retry loops. The application delegates all network I/O to the local agent.
• Unmatched Infrastructure Context: Running directly on the host node enables unified correlation between application telemetry, container stdout logs, and hardware performance metrics (hostmetrics).
• Local Buffer Resilience: If the wider cluster network or external internet experiences a transient interruption, the local agent continues buffering telemetry locally on the node without stalling application execution.

Limitations of the Agent Pattern

• Coupled Resource Scaling: In a DaemonSet deployment, the Collector's memory and CPU allocations are tied to worker node counts rather than telemetry data volume. High-traffic nodes running telemetry-heavy services may experience memory pressure, while low-traffic nodes waste allocated memory limits.
• Sidecar Resource Bloat: If deployed as a sidecar across 2,000 application pods, the cluster runs 2,000 independent Collector processes, each with its own baseline memory and CPU, which adds up to a large amount of duplicated overhead.
• Broad Attack Surface for Secrets: If the Agent must export directly to an external vendor, secret API tokens and TLS client certificates must be distributed to every node or sidecar across the cluster, violating security best practices.
• Inability to Perform Cluster-Wide State Operations: Local agents have visibility only into telemetry emitted by workloads running on their specific node. They cannot perform cluster-wide tail-based sampling or cross-service metric aggregations because spans of a distributed trace are scattered across multiple nodes.

---

The Gateway Pattern (Centralized Standalone Tier)

In the Gateway pattern, the OpenTelemetry Collector is deployed as a centralized, standalone service tier. In Kubernetes, this is managed as a standard horizontal Deployment fronted by a Kubernetes Service (ClusterIP), an internal Layer 4 Network Load Balancer (NLB), or an Envoy/Ingress gateway.

[App Pods / Microservices]
[Distributed across Nodes]
            │
            │ OTLP (gRPC / HTTP)
            ▼
+───────────────────────────────────────────────────────────+
|               Internal Load Balancer / VIP                |
+───────────────────────────────────────────────────────────+
            │
            ├──────────────────────┬──────────────────────┐
            ▼                      ▼                      ▼
+──────────────────────+ +──────────────────────+ +──────────────────────+
|  Gateway Replica 1   | |  Gateway Replica 2   | |  Gateway Replica 3   |
|  (Deployment Pod)    | |  (Deployment Pod)    | |  (Deployment Pod)    |
|                      | |                      | |                      |
|  - Central Secrets   | |  - Central Secrets   | |  - Central Secrets   |
|  - OTTL PII Redaction| |  - OTTL PII Redaction| |  - OTTL PII Redaction|
|  - Tail Sampling     | |  - Tail Sampling     | |  - Tail Sampling     |
|  - Heavy Batching    | |  - Heavy Batching    | |  - Heavy Batching    |
+──────────────────────+ +──────────────────────+ +──────────────────────+
            │                      │                      │
            └──────────────────────┼──────────────────────┘
                                   │ Single Hardened Egress
                                   ▼
                  +─────────────────────────────────+
                  | Observability Backends / SaaS   |
                  | (Prometheus, Jaeger, Cloud APM) |
                  +─────────────────────────────────+

Operational Responsibilities of the Gateway

1. Centralized Credential Management: Sensitive observability backend credentials—such as Datadog API keys, Dynatrace tokens, Google Cloud service account JSON keys, and AWS IAM roles—are mounted exclusively into the Gateway deployment. Application developers and node-level DaemonSets never have access to backend secrets.
2. Egress Traffic Consolidation: The Gateway tier acts as a single, hardened network exit point. Security teams can restrict outbound firewall and NAT gateway rules so that only the Gateway tier's CIDR blocks or service accounts are permitted to send outbound traffic to external internet endpoints over port 443.
3. Data Governance & PII Scrubbing: The Gateway executes centralized OpenTelemetry Transformation Language (OTTL) rules. It strips Authorization headers, masks credit card patterns, redacts email addresses, drops debug spans from production namespaces, and normalizes semantic conventions across all teams.
4. Stateful Pipeline Execution: Because the Gateway runs as a consolidated cluster, it can host stateful processors such as the tail_sampling processor (evaluating full trace error states), the spanmetrics connector (calculating golden signals from spans), and metric deduplication.
5. Batching and Egress Optimization: Gateways consolidate telemetry from hundreds of upstream services, grouping data into large, highly compressed batches (ZSTD or GZIP). This minimizes outbound HTTP/gRPC request overhead, optimizes network MTU utilization, and significantly reduces SaaS ingestion bills.

Strengths of the Gateway Pattern

• Strict Security Perimeter: Zero credential leakage into developer namespaces or node agents; centralized audit logging and strict egress filtering.
• Independent Elastic Scaling: The Gateway deployment scales horizontally via the Horizontal Pod Autoscaler (HPA) based strictly on telemetry ingress throughput (CPU, memory, refused spans), completely decoupled from the number of Kubernetes worker nodes.
• Central Policy Control: Observability platform teams can modify retention policies, change vendor destinations, add secondary cold-storage S3 exports, or adjust sampling ratios in a single Gateway configuration without restarting application pods.

Limitations of the Gateway Pattern

• Additional Network Hop: Telemetry travels from application pods over the internal cluster network to reach the Gateway pods, introducing a network hop and consuming inter-pod bandwidth.
• Application Buffering Risk: If the central Gateway cluster suffers an outage or network partition, application SDKs sending directly to the Gateway must buffer telemetry in memory. If SDK queues fill up, applications will drop telemetry locally.
• No Direct Host Visibility: A central Gateway cannot inspect /proc or /sys on remote application worker nodes and cannot tail local container log files from /var/log/pods.

---

The Hybrid Two-Tier Pattern (Production Best Practice)

In large-scale production Kubernetes and multi-cloud environments, the Hybrid Two-Tier Pattern is a common choice; the OpenTelemetry documentation describes combining agents with gateways in exactly this way. It combines the low-latency, node-level visibility of the Agent pattern with the centralized security, governance, and stateful capabilities of the Gateway pattern.

============================== TIER 1: NODE AGENT ==============================
  Workload Pods (App SDKs) ----> HostIP:4317 -------> Collector DaemonSet
                                                       - hostmetrics (/proc)
                                                       - filelog (/var/log/pods)
                                                       - k8sattributes (pod IP)
                                                       - minimal memory buffer
                                                              │
============================== INTER-TIER ROUTING ==============================
                                                              ▼
                                              Internal Load Balancing Exporter
                                              (Consistent Hashing on TraceId)
                                                              │
============================== TIER 2: CENTRAL GATEWAY =========================
                                                              ▼
                                                  Collector Gateway Deployment
                                                  - Tail-based sampling
                                                  - OTTL PII redaction
                                                  - Secret credentials & mTLS
                                                  - Multi-destination routing
                                                              │
============================== EGRESS DESTINATIONS ============================
                                                              ▼
                                    [Prometheus]  [Tempo/Jaeger]  [Elastic/S3]

Division of Labor in a Two-Tier Architecture

Why Two-Tier is the Gold Standard

1. Minimal Client Buffering: Applications offload to the Tier 1 node agent over a short node-local hop, keeping SDK queues small.
2. Perfect Infrastructure Enrichment: The Tier 1 agent injects exact node, pod, and container metadata at the point of origin, where network mapping is fast and unambiguous.
3. Rock-Solid Security: All sensitive vendor API keys reside solely within the Tier 2 Gateway deployment. Compromising an application pod or worker node does not reveal production observability secrets.
4. Massive Cost Optimization: The Tier 2 Gateway performs global tail-based sampling (e.g., dropping 90% of boring HTTP 200 traces while preserving 100% of HTTP 500 errors) and aggregates spans into metrics before sending data to expensive SaaS vendors.

---

Comparison Matrix: Collector Deployment Topologies

---

Summary of Architectural Guidelines

• Use the Agent (DaemonSet) Pattern when you need unified host metrics, container log tailing, and local pod attribute enrichment in a self-hosted Kubernetes cluster with homogeneous workloads.
• Use the Agent (Sidecar) Pattern when deploying to serverless container platforms (such as AWS Fargate, Google Cloud Run, or Azure Container Apps) where node-level access is prohibited, or where strict multi-tenant network isolation requires zero shared processes.
• Use the Gateway (Deployment) Pattern when your primary goal is centralizing egress network traffic, securely managing vendor API tokens, enforcing enterprise data governance with OTTL, and running stateful tail-based sampling.
• Use the Hybrid Two-Tier Pattern for enterprise production environments to combine low-latency local offloading and metadata enrichment with centralized egress security, elastic scaling, and intelligent data reduction.