15.2 Application Insights & Specialized Azure Monitor Insights

15.2 Application Insights & Specialized Azure Monitor Insights

The agent and data collection rules get platform and guest telemetry into the workspace. Application Insights and the specialized Insights experiences are what turn that raw telemetry into application-level and workload-level answers.

1. Application Insights: Deep Application Performance Management (APM)

Application Insights is the application performance management (APM) feature of Azure Monitor. It instruments running code to monitor live application health, identify latency bottlenecks, trace distributed microservice calls, and track business conversion flows.

Ingestion Connection Strings vs. Legacy Instrumentation Keys

Earlier iterations of Application Insights relied on a static GUID called the Instrumentation Key (iKey). The iKey is deprecated because it lacked regional ingestion targeting and authenticated access controls. Modern Application Insights requires an Ingestion Connection String, which encapsulates the target ingestion endpoint, regional routing, and authorization credentials:

InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/

Codeless / Auto-Instrumentation vs. SDK-Based Instrumentation

A primary architectural decision tested on the AZ-400 exam is choosing between codeless auto-instrumentation and code-based SDK instrumentation.

Telemetry Data Model

Application Insights stores telemetry in dedicated tables within its underlying Log Analytics workspace:

• requests: Inbound HTTP/gRPC requests, including URL, HTTP method, response duration, status code (200, 404, 500), and success boolean.
• dependencies: Outbound network interactions, such as SQL database calls, HTTP REST calls to downstream services, Azure Service Bus queues, or Azure Blob storage requests.
• exceptions: Unhandled runtime crashes, caught exceptions explicitly passed to telemetry, call stacks, and inner exceptions.
• traces: In-process diagnostic logging generated through framework loggers (ILogger, log4j, winston, Serilog).
• pageViews: Client-side single-page application (SPA) navigation and page load timings captured via the Application Insights JavaScript SDK.
• customEvents & customMetrics: Explicit business telemetry emitted via TelemetryClient.TrackEvent() or TelemetryClient.TrackMetric().

Code Example: SDK Custom Enrichment with TelemetryInitializer

When running in enterprise multi-tenant environments, DevOps engineers enrich all outgoing telemetry with deployment and tenant metadata:

using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;

// Custom TelemetryInitializer runs on EVERY telemetry item before transmission
public class DeploymentTelemetryInitializer : ITelemetryInitializer
{
    private readonly string _deploymentRing;
    private readonly string _gitCommitHash;

    public DeploymentTelemetryInitializer(string ring, string commit)
    { 
        _deploymentRing = ring;
        _gitCommitHash = commit;
    }

    public void Initialize(ITelemetry telemetry)
    {
        // Enrich telemetry with deployment ring (Ring0, Ring1) and Git SHA
        if (!telemetry.Context.GlobalProperties.ContainsKey("DeploymentRing"))
        {
            telemetry.Context.GlobalProperties.Add("DeploymentRing", _deploymentRing);
            telemetry.Context.GlobalProperties.Add("GitCommit", _gitCommitHash);
        }
    }
}

Application Insights Sampling Strategies

High-traffic enterprise applications can generate hundreds of gigabytes of telemetry daily. To reduce cost and prevent network throttling, Application Insights provides three sampling mechanisms:

1. Adaptive Sampling: The default mechanism for .NET and ASP.NET Core SDKs. Dynamically adjusts the sampling percentage based on current traffic volume. If traffic surges, sampling drops to keep transmission within configured rates. Crucially, adaptive sampling ensures related telemetry (a request, its downstream SQL dependencies, and resulting exceptions) are retained or dropped together as a coherent trace.
2. Fixed-Rate Sampling: Configured statically in application configuration (e.g., sample exactly 25% of all traffic). Operates consistently regardless of volume.
3. Ingestion Sampling: Applied at the Azure Monitor ingestion endpoint after data arrives in the cloud. It drops incoming records that exceed rate limits. Exam Warning: Ingestion sampling drops items randomly without preserving cross-telemetry correlation, and the network bandwidth from host to cloud has already been consumed. Use SDK-level adaptive sampling instead!

---

2. Specialized Azure Monitor Insights

Beyond core metrics and logs, Azure Monitor provides turnkey, curated domain monitoring solutions called Insights.

                        [Azure Monitor Insights]
       ┌───────────────────┼───────────────────┬───────────────────┐
       ▼                   ▼                   ▼                   ▼
  [VM Insights]   [Container Insights]   [Storage Insights]  [Network Insights]
  • Guest OS metrics  • AKS cluster health    • Transaction latency  • Traffic Analytics
  • Service Map       • Pod CPU/Memory limits • Ingress/Egress MB    • NSG Flow Logs
  • TCP Dependencies  • Prometheus scrape     • Capacity & Throttling• Connection Monitor

1. Azure VM Insights

VM Insights monitors the performance and health of virtual machines and virtual machine scale sets.

• Performance: Pre-built workbooks analyzing processor utilization, memory pressure, disk read/write IOPS, and network throughput.
• Map (Service Map): Deployed as a dependency agent extension alongside AMA. Discovers running processes and active TCP network connections across servers in real time. For DevOps teams executing cloud migrations or debugging cascading failures, Service Map reveals upstream callers, downstream database dependencies, and failed connection attempts across ports.

2. Azure Container Insights

Container Insights monitors Azure Kubernetes Service (AKS), Azure Arc-enabled Kubernetes, and Azure Container Instances.

• Architecture: Deployed as a specialized containerized Azure Monitor Agent daemonset (ama-logs) running on every cluster node.
• Telemetry Gathered: CPU and memory utilization against Kubernetes resource requests and limits (KubeNodeInventory, KubePodInventory), pod lifecycle crash loops (CrashLoopBackOff), and container stdout/stderr console logs.
• Prometheus Metric Collection: Container Insights includes a managed Prometheus scraping engine. It automatically scrapes Prometheus-formatted metric endpoints exposed by Kubernetes pods (e.g., /metrics) and stores them in Azure Monitor Managed Service for Prometheus without requiring a self-hosted Prometheus/Thanos cluster.

3. Azure Storage & Network Insights

• Storage Insights: Tracks end-to-end transaction latency, server latency, blob capacity, and HTTP 4xx/5xx authorization and throttling errors across Blob, File, Table, and Queue services.
• Network Insights & Traffic Analytics: Analyzes Network Security Group (NSG) flow logs and Virtual Network flow logs. Traffic Analytics decodes IP flows to identify malicious traffic, geographic traffic distribution, internal subnet bottlenecks, and cross-region egress costs.

---

3. Diagnostic Settings: Routing Resource Logs and Metrics

Every native Azure resource (Key Vaults, Cosmos DB, Application Gateways, AKS clusters, Azure SQL) emits two streams of operational data:

1. Resource Metrics: Numerical metrics emitted automatically to Azure Monitor Metrics.
2. Resource Logs (formerly Diagnostic Logs): Detailed internal event logs (e.g., Key Vault secret access audit logs, Application Gateway access logs, firewall rule evaluations).

To capture and retain Resource Logs, DevOps engineers must configure Diagnostic Settings.

                            [Azure PaaS / IaaS Resource]
                            (e.g., Azure Key Vault / AKS)
                                          │
                            [Diagnostic Settings Engine]
                                          │
                   ┌──────────────────────┼──────────────────────┐
                   ▼                      ▼                      ▼
        [Log Analytics Workspace]  [Azure Storage Account]   [Azure Event Hub]
        • KQL interactive queries  • Long-term compliance    • Real-time streaming
        • Cross-resource joins     • 1-to-10 year archival   • Third-party SIEM
        • Sentinel / Alert rules   • Lowest storage cost     • Splunk / Datadog

Destination Routing Comparison

Azure CLI Implementation

# Configure Diagnostic Setting on Azure Key Vault to route audit logs to Log Analytics and Event Hub
az monitor diagnostic-settings create \
  --name "diag-keyvault-audit" \
  --resource "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-security/providers/Microsoft.KeyVault/vaults/kv-prod-secrets" \
  --workspace "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-monitor/providers/Microsoft.OperationalInsights/workspaces/law-central-prod" \
  --event-hub "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-monitor/providers/Microsoft.EventHub/namespaces/eh-siem-stream/eventhubs/hub-security-logs" \
  --event-hub-rule "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-monitor/providers/Microsoft.EventHub/namespaces/eh-siem-stream/authorizationrules/RootManageSharedAccessKey" \
  --logs '[{"categoryGroup":"audit","enabled":true,"retentionPolicy":{"days":0,"enabled":false}}]' \
  --metrics '[{"category":"AllMetrics","enabled":true,"retentionPolicy":{"days":0,"enabled":false}}]'

---

4. Telemetry Collection Options Matrix

---

5. Realistic Exam Scenario & Common Traps

Scenario: High-Volume Fintech Regulatory Observability

Organization: Global Trade Financial runs an electronic settlement platform comprising 150 Azure VMs, a 20-node AKS cluster, and Azure SQL databases.

• Constraint 1: The internal SOC requires all virtual machine failed authentication attempts (Event ID 4625) and all Azure Key Vault audit events to be analyzed in real time. Informational successful logon events (Event ID 4624) must be excluded at the machine boundary to avoid blowing through their 500 GB/day Log Analytics ingestion quota.
• Constraint 2: Corporate compliance requires all transaction audit logs to be retained for 7 years at minimum financial cost.
• Constraint 3: The site reliability engineering team needs to ingest Prometheus metrics from their payment container pods without managing a dedicated Prometheus cluster.

DevOps Architect Solution:

1. Deploy the Azure Monitor Agent (AMA) across all virtual machines and bind them to a Data Collection Rule (DCR) with a KQL stream transform: source | where EventID == 4625. This drops Event 4624 before network transmission, satisfying the ingestion budget.
2. Configure Diagnostic Settings on the Azure Key Vault and transaction databases targeting an Azure Storage Account with a Lifecycle Management policy transitioning blobs to Cold tier after 30 days and Archive tier after 90 days (7-year retention).
3. Enable Azure Container Insights on the AKS cluster with managed Prometheus scraping enabled, streaming pod metrics directly into Azure Monitor Managed Service for Prometheus.

Common Exam Traps to Avoid

• Trap: Selecting the legacy Log Analytics agent (MMA/OMS) in any design. The legacy agent was retired in August 2024. The correct choice on modern AZ-400 questions is always Azure Monitor Agent (AMA) with Data Collection Rules.
• Trap: Selecting Log Analytics storage for 7-year compliance archives. While Log Analytics supports up to 730 days interactive retention and 12-year archive, storing high-volume raw compliance logs in an Azure Storage Account via Diagnostic Settings is drastically cheaper and the standard exam answer for multi-year regulatory retention.
• Trap: Choosing codeless auto-instrumentation when custom telemetry properties are required. If a scenario requires tracking custom business metrics or injecting context properties (such as user tenant IDs or checkout transaction dollar values), the answer is always SDK-based instrumentation with custom TelemetryInitializer or TelemetryClient.
• Trap: Attempting to stream logs to Splunk or Datadog directly from Log Analytics. Azure resources do not push directly from Log Analytics to third-party SIEMs; instead, configure Diagnostic Settings with an Azure Event Hub destination, which acts as the high-throughput streaming bridge to external SIEM connectors.