3.3 Cloud-Native Deployment: Dynatrace Operator on Kubernetes & OpenShift

Cloud-Native Deployment: Dynatrace Operator on Kubernetes & OpenShift

Exam Focus: The Dynatrace Certified Associate exam places substantial emphasis on cloud-native deployment patterns, specifically the architecture of the Dynatrace Operator, the differences between the four primary DynaKube deployment modes, the operation of the Mutating Webhook admission controller, and the role of the Container Storage Interface (CSI) driver.

In modern container orchestration environments like Kubernetes, Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), Microsoft Azure Kubernetes Service (AKS), and Red Hat OpenShift, managing observability through traditional, static DaemonSets or manually altered Dockerfiles is anti-pattern. Ephemeral container lifecycles, autoscaling node groups, non-root security constraints, and immutable filesystems require an automated, declarative management approach.

The Dynatrace Operator embodies the official Kubernetes Operator pattern. It acts as an automated, in-cluster Site Reliability Engineer (SRE) that continuously reconciles the desired state of monitoring components against the cluster's actual state.

---

The Dynatrace Operator Architecture and the DynaKube Custom Resource

The Dynatrace Operator registers a Custom Resource Definition (CRD) called DynaKube (apiGroup: core.dynatrace.com). When a cluster administrator applies a DynaKube manifest, the Operator deploys, configures, and manages several specialized architectural components:

1. Operator Controller Manager (dynatrace-operator): The core control loop that reconciles DynaKube custom resources, coordinates agent updates, configures RBAC policies, and provisions routing to Dynatrace ActiveGates.
2. Mutating Webhook Controller (dynatrace-webhook): An admission controller registered with the Kubernetes API server (kube-apiserver). When pods are created in monitored namespaces, the webhook intercepts the pod creation requests, mutates the pod specifications, and injects the necessary environment variables and volume mounts required for OneAgent instrumentation.
3. CSI Driver DaemonSet (dynatrace-oneagent-csi-driver): A specialized Container Storage Interface (CSI) driver deployed across worker nodes. It downloads and caches OneAgent binaries once on the host node storage and binds them as read-only volumes into monitored application pods.
4. Host Monitoring DaemonSet: A lightweight node daemon responsible for gathering node-level infrastructure metrics, cgroup resource consumption, and container engine health.

---

Comprehensive Comparison of DynaKube Deployment Modes

A central focus of the certification exam is selecting the appropriate deployment mode within the DynaKube specification. Dynatrace provides four primary modes, each engineered for specific infrastructure topologies and security postures.

Mode Details

• CloudNativeFullStack: The official Dynatrace recommended standard for modern Kubernetes and OpenShift. It completely decouples node-level infrastructure monitoring from application runtime injection. Application pods are instrumented cleanly at admission time using the CSI driver, eliminating the need for privileged container runtime injection on the node.
• ClassicFullStack: Mirrors traditional VM-based OneAgent deployment. A single privileged OneAgent container runs per node as a DaemonSet, discovers container runtimes via local sockets, and injects code modules into container processes from the host. This mode is used when clusters lack CSI driver support or when migrating existing VM environments to containerized architectures.
• ApplicationMonitoring: Specifically engineered for serverless or tightly restricted environments where cluster administrators do not manage the underlying virtual machine worker nodes (such as AWS Fargate for EKS). Because DaemonSets cannot be scheduled on Fargate, this mode relies strictly on the Mutating Webhook to inject OneAgent into application containers at the pod level.
• HostMonitoring: Deploys a DaemonSet focused strictly on collecting host node CPU, memory, storage, network, and container cgroup metrics. It does not inject code modules into application containers and does not capture PurePaths or database transactions. It consumes lower-cost Infrastructure Monitoring licensing.

---

The Mutating Webhook Admission Controller Deep Dive

To understand how OneAgent instruments containerized workloads in CloudNativeFullStack and ApplicationMonitoring modes, examine the step-by-step admission review lifecycle:

1. Developer Action    ==> kubectl apply -f deployment.yaml
2. API Server Review   ==> kube-apiserver receives Pod creation request
3. Webhook Dispatch    ==> AdmissionReview sent to dynatrace-webhook service
4. Namespace Filter    ==> Webhook checks namespace labels (e.g., oneagent.dynatrace.com/inject)
5. Pod Mutation        ==> Webhook patches Pod spec: adds init-container, CSI mount, env vars
6. Node Scheduling     ==> kube-scheduler assigns mutated Pod to worker node
7. CSI Volume Attach   ==> CSI driver mounts cached agent binaries into /opt/dynatrace/oneagent-paas
8. Init Execution      ==> install-oneagent container verifies configuration and certificates
9. App Container Start ==> Application boots with LD_PRELOAD loading OneAgent into memory

Namespace Targeting and Opt-In / Opt-Out Mechanics

The Dynatrace Mutating Webhook does not blindly inject every pod across the entire cluster. By default, it inspects Kubernetes namespace labels. Administrators control injection scope through labeling:

# Enable automatic injection for a namespace
kubectl label namespace production-ecommerce oneagent.dynatrace.com/inject=true

# Disable injection explicitly for a namespace
kubectl label namespace system-maintenance oneagent.dynatrace.com/inject=false

Individual workloads can override namespace defaults using pod annotations in their deployment manifests:

metadata:
  annotations:
    oneagent.dynatrace.com/inject: "false" # Opts this specific deployment out

---

The Container Storage Interface (CSI) Driver Architecture

Prior to the introduction of the Dynatrace CSI driver, monitoring containerized applications required either baking OneAgent binaries into every base Docker image (causing massive image bloat) or executing an init container that downloaded the 200MB+ OneAgent installer from Dynatrace SaaS every time a pod scaled up. During rapid autoscaling events or node failovers, dozens of pods simultaneously downloading installer packages would exhaust network bandwidth and delay pod startup times by minutes.

The dynatrace-oneagent-csi-driver solves this problem permanently:

• It runs as a lightweight DaemonSet on each worker node.
• It connects to the Dynatrace cluster, downloads the required OneAgent code modules once, and stores them in a local cache on the host node storage.
• When a newly scheduled application pod is mutated by the webhook, a CSI volume is attached to the pod.
• The CSI driver binds the pre-cached OneAgent binaries directly into the application container's filesystem as a read-only volume mounted at /opt/dynatrace/oneagent-paas.
• Container startup is nearly instantaneous (sub-second), image bloat is completely eliminated, and external egress traffic is reduced by orders of magnitude.

---

Red Hat OpenShift Integration and Security Context Constraints (SCC)

Red Hat OpenShift enforces stringent security boundaries through Security Context Constraints (SCC). By default, OpenShift prohibits pods from running as the root user, mounting host filesystems, or accessing host namespaces.

The Dynatrace Operator is fully certified for OpenShift and available directly through the OpenShift OperatorHub. During deployment, the Operator automatically creates and configures the required SCC bindings:

• The CSI driver and node-monitoring daemonsets are granted the necessary privileged or custom SCCs required to mount storage volumes and inspect node cgroups.
• Monitored application pods remain completely unprivileged, running under OpenShift's standard restricted-v2 SCC without violating enterprise compliance.
• The Operator seamlessly configures OpenShift Routes for secure ActiveGate communication across clusters.

---

Cluster Verification, Lifecycle Upgrades & Troubleshooting

Administering the Dynatrace Operator requires proficiency with diagnostic and status verification commands:

Verifying Deployment Health

# Check the status of the DynaKube custom resource
kubectl get dynakube -n dynatrace

# Detailed inspection of DynaKube conditions, phase, and agent version
kubectl describe dynakube <dynakube-name> -n dynatrace

# Verify all operator, webhook, and CSI pods are running
kubectl get pods -n dynatrace -o wide

Troubleshooting Mutating Webhook Injection Failures

If application pods in a target namespace are running but lack OneAgent code modules (no PurePaths or services appearing in Dynatrace):

1. Verify Namespace Labeling: Confirm that the namespace contains the exact label oneagent.dynatrace.com/inject=true using kubectl get ns --show-labels.
2. Check Webhook Configuration: Verify that the mutating webhook configuration is registered and active:

   kubectl get mutatingwebhookconfiguration dynatrace-webhook -o yaml
   

3. Inspect Pod Init Container Logs: If the webhook mutated the pod but injection failed during startup, inspect the OneAgent init container output:

   kubectl logs <pod-name> -c install-oneagent -n <application-namespace>
   

4. Review Operator Logs: Inspect the central operator controller logs for admission review rejections, certificate synchronization issues, or ActiveGate routing errors:

   kubectl logs -n dynatrace -l app.kubernetes.io/name=dynatrace-operator --tail=200