12.2 Runtime Manager: Application Management, Alerts, Insights & Logs

Runtime Manager: Application Management, Alerts, Insights & Logs

Once Mule applications are designed and tested, Anypoint Runtime Manager serves as the unified administrative control plane for deploying, configuring, monitoring, and troubleshooting integrations across cloud, on-premises, and hybrid environments. This section explores deployment automation with the Mule Maven Plugin, application property management, zero-downtime rolling updates, alert configurations, log retention, and real-time telemetry with Anypoint Monitoring.

---

1. Runtime Manager Deployment Mechanisms

Runtime Manager provides multiple deployment mechanisms to support both ad-hoc administrative deployments and fully automated Continuous Integration / Continuous Deployment (CI/CD) pipelines.

+-----------------------------------------------------------------------------------------+
|                        RUNTIME MANAGER DEPLOYMENT PIPELINE                              |
|                                                                                         |
|   [DEVELOPER / STUDIO]            [GIT REPOSITORY]            [CI/CD PIPELINE]          |
|   - Source Code                   - Version Control           - Jenkins / GitHub Actions|
|   - pom.xml Configuration         - Pull Request Merge        - Automated Build & Test  |
|            |                                                         |                  |
|            +----------------------------+----------------------------+                  |
|                                         |                                               |
|                                         v                                               |
|                     +---------------------------------------+                           |
|                     | MULE MAVEN PLUGIN (mule-maven-plugin) |                           |
|                     | mvn clean deploy -DmuleDeploy         |                           |
|                     +---------------------------------------+                           |
|                                         |                                               |
|                                         v                                               |
|                     +---------------------------------------+                           |
|                     | ANYPOINT PLATFORM REST API            |                           |
|                     +---------------------------------------+                           |
|                        /                |                \                              |
|                       v                 v                 v                             |
|                [CloudHub 1.0]    [CloudHub 2.0]    [Runtime Fabric]                     |
|                (AWS EC2 VMs)     (Hyperforce K8s)  (Self-Managed K8s)                   |
+-----------------------------------------------------------------------------------------+

Deployment Methods

1. Anypoint Platform Web Console: Interactive UI in Runtime Manager for uploading packaged .jar files, setting worker sizing, and configuring environment properties.
2. Anypoint CLI (Command Line Interface): Scriptable command-line tool (anypoint-cli runtime-mgr cloudhub-application deploy ...) for shell-based automation.
3. Mule Maven Plugin (mule-maven-plugin): The enterprise standard for CI/CD automation. Configured in pom.xml, it packages the application, runs MUnit tests, and uploads the artifact directly to Runtime Manager via the Anypoint Platform deployment APIs.

Configuring the Mule Maven Plugin for CloudHub

To automate deployments via Maven, add the <cloudhubDeployment> configuration block to the mule-maven-plugin declaration in pom.xml:

<plugin>
    <groupId>org.mule.tools.maven</groupId>
    <artifactId>mule-maven-plugin</artifactId>
    <version>3.8.2</version>
    <extensions>true</extensions>
    <configuration>
        <cloudhubDeployment>
            <uri>https://anypoint.mulesoft.com</uri>
            <muleVersion>4.4.0</muleVersion>
            <username>${env.ANYPOINT_USERNAME}</username>
            <password>${env.ANYPOINT_PASSWORD}</password>
            <environment>Production</environment>
            <applicationName>acme-orders-prc-api-v1</applicationName>
            <businessGroup>Acme Global Integration</businessGroup>
            <workerType>MICRO</workerType> <!-- MICRO = 0.1 vCore, SMALL = 0.2 vCore, MEDIUM = 1 vCore -->
            <workers>2</workers>
            <region>us-east-1</region>
            <objectStoreV2>true</objectStoreV2>
            <properties>
                <env>prod</env>
                <api.id>18920412</api.id>
                <anypoint.platform.analytics_base_uri>https://analytics-ingest.anypoint.mulesoft.com</anypoint.platform.analytics_base_uri>
            </properties>
            <secureProperties>
                <db.password>${env.DB_PASSWORD}</db.password>
                <encryption.key>${env.ENCRYPTION_KEY}</encryption.key>
            </secureProperties>
        </cloudhubDeployment>
    </configuration>
</plugin>

To execute the automated build and deployment command from a CI/CD pipeline, run:

mvn clean deploy -DmuleDeploy -Denv.ANYPOINT_USERNAME=ci_user -Denv.ANYPOINT_PASSWORD=ci_token

---

2. Application Property Management & Runtime Hierarchy

Mule 4 applications load configuration properties from multiple sources. Understanding property precedence is crucial when deploying across development, staging, and production environments.

+-----------------------------------------------------------------------------------------+
|                         PROPERTY RESOLUTION PRECEDENCE (HIERARCHY)                      |
|                                                                                         |
|   [HIGHEST PRIORITY]                                                                    |
|   1. Java System Properties (-Ddb.port=3306 passed at runtime startup)                  |
|          |                                                                              |
|          v                                                                              |
|   2. Runtime Manager Application Properties (Configured in UI / Maven Deployment Block) |
|          |                                                                              |
|          v                                                                              |
|   3. Secure Configuration Properties (secure::db.password encrypted via Secure Tool)    |
|          |                                                                              |
|          v                                                                              |
|   4. Environment-Specific YAML/Property Files (e.g., config-prod.yaml in JAR classpath) |
|          |                                                                              |
|          v                                                                              |
|   5. Default Base Properties (config-base.yaml or mule-app.properties)                  |
|   [LOWEST PRIORITY]                                                                     |
+-----------------------------------------------------------------------------------------+

Key Property Management Rules:

• Runtime Overrides: Properties defined in the Runtime Manager Properties tab take precedence over property files packaged inside the deployable .jar file.
• Protecting Secrets: Sensitive credentials (passwords, private API keys, client secrets) must be encrypted using the Mule Secure Properties Tool and referenced using secure::property.name syntax. In Runtime Manager, prefixing property names or setting hidden flags prevents secret values from being displayed in plain text in the web console.

---

3. Zero-Downtime Rolling Deployments

When updating an existing CloudHub application (e.g., deploying bug fixes, new features, or changing worker sizing), CloudHub executes a zero-downtime rolling update.

+-----------------------------------------------------------------------------------------+
|                        CLOUDHUB ZERO-DOWNTIME ROLLING DEPLOYMENT                        |
|                                                                                         |
|   STEP 1: INITIAL STATE                                                                 |
|   [Load Balancer] ---> [Worker 1 (v1.0.0) - ACTIVE]                                     |
|                                                                                         |
|   STEP 2: PROVISION & START NEW WORKER                                                  |
|   [Load Balancer] ---> [Worker 1 (v1.0.0) - ACTIVE]                                     |
|                        [Worker 2 (v2.0.0) - STARTING & HEALTH CHECK]                    |
|                                                                                         |
|   STEP 3: TRAFFIC CUTOVER (Health Check Passes)                                         |
|   [Load Balancer]                                                                       |
|          |                                                                              |
|          +-----------> [Worker 2 (v2.0.0) - ACTIVE]                                     |
|                        [Worker 1 (v1.0.0) - DRAINING IN-FLIGHT REQUESTS]                |
|                                                                                         |
|   STEP 4: OLD WORKER TERMINATION                                                        |
|   [Load Balancer] ---> [Worker 2 (v2.0.0) - ACTIVE]                                     |
|                        (Worker 1 Terminated - Zero Dropped Requests)                    |
+-----------------------------------------------------------------------------------------+

The 5 Phases of a Rolling Update:

1. Instance Provisioning: CloudHub provisions a new EC2 worker instance alongside the currently running worker.
2. Application Extraction & Deployment: The new application JAR is downloaded and initialized on the new worker.
3. Health Check & Flow Startup: CloudHub verifies that all Mule flows, HTTP Listeners, and database connection pools initialize successfully.
4. Load Balancer Cutover: The load balancer updates its routing table, redirecting new incoming client requests to the new worker.
5. Graceful Draining & Decommissioning: The legacy worker finishes processing all active in-flight requests and is cleanly terminated.

[!IMPORTANT] Failed Deployment Safety Net If the new application version fails to start (e.g., due to an invalid database password or XML syntax error), CloudHub aborts the update, preserves the original working instance, and leaves existing traffic uninterrupted.

---

4. Alerting Framework: Standard vs. Custom Alerts

Runtime Manager allows administrators to configure automated email notifications based on operational events.

+-----------------------------------------------------------------------------------------+
|                        RUNTIME MANAGER ALERTING ARCHITECTURE                            |
|                                                                                         |
|   +---------------------------------------------------------------------------------+   |
|   | STANDARD ALERTS (Infrastructure Telemetry)                                      |   |
|   | - Worker Unresponsive (Heartbeat Timeout)                                       |   |
|   | - CPU Utilization > 80% for 5 Minutes                                           |   |
|   | - Memory Utilization > 80%                                                      |   |
|   | - Application Deployment Failed / Succeeded                                     |   |
|   +---------------------------------------------------------------------------------+   |
|                                                                                         |
|   +---------------------------------------------------------------------------------+   |
|   | CUSTOM ALERTS (Application Business Events)                                     |   |
|   |                                                                                 |   |
|   |   Mule Flow Catch Handler:                                                      |   |
|   |   <cloudhub:create-notification priority="ERROR">                               |   |
|   |       Order Payment Gateway Connectivity Failed for ID #[vars.orderId]          |   |
|   |   </cloudhub:create-notification>                                               |   |
|   |                  |                                                              |   |
|   |                  v                                                              |   |
|   |   Runtime Manager Custom Alert Rule:                                            |   |
|   |   Trigger on Custom Notification containing 'Payment Gateway' -> Email Ops Team |   |
|   +---------------------------------------------------------------------------------+   |
+-----------------------------------------------------------------------------------------+

Standard Alerts (Out-of-the-Box)

Standard alerts monitor infrastructure and deployment health without requiring changes to Mule application code:

• Worker Unresponsive: Triggered when the CloudHub monitoring daemon fails to receive a heartbeat ping from the worker JVM.
• CPU Utilization: Triggered when CPU consumption exceeds a specified percentage (e.g., > 80%) for a consecutive time threshold.
• Memory Utilization: Triggered when JVM heap usage exceeds a configured threshold.
• Deployment State Changes: Triggered on deployment success or deployment failure.

Custom Alerts via CloudHub Connector

Custom alerts enable Mule flows to trigger notifications based on business exceptions, validation errors, or security anomalies using the CloudHub Connector (<cloudhub:create-notification>):

<flow name="payment-processing-flow" doc:name="Payment Processing Flow">
    <http:listener config-ref="HTTP_Listener_config" path="/pay" doc:name="Listener"/>
    
    <try doc:name="Try Process Payment">
        <http:request config-ref="Stripe_HTTP_Config" path="/charges" method="POST" doc:name="Charge Card"/>
        <error-handler>
            <on-error-continue type="HTTP:CONNECTIVITY" doc:name="Catch Payment Error">
                <logger level="ERROR" message="#['Payment service unreachable for Order: ' ++ vars.orderId]" doc:name="Log Error"/>
                
                <!-- Trigger CloudHub Notification -->
                <cloudhub:create-notification 
                    config-ref="CloudHub_Config" 
                    domain="#[app.name]" 
                    priority="ERROR" 
                    doc:name="Send Alert Notification">
                    <cloudhub:message><![CDATA[#[%dw 2.0
output text/plain
---
"CRITICAL: Payment gateway unreachable for Order ID: " ++ (vars.orderId default 'UNKNOWN') ++ ". Reason: " ++ error.description]]]></cloudhub:message>
                </cloudhub:create-notification>
            </on-error-continue>
        </error-handler>
    </try>
</flow>

---

5. Log Management & External Forwarding

Runtime Manager captures console output (stdout, stderr, and Logger component messages) in real time.

CloudHub Log Retention Policy

• Retention Threshold: CloudHub stores logs up to 30 days or 100 MB per application, whichever limit is reached first.
• Log Rollover: In high-throughput applications generating megabytes of logs per hour, older logs are automatically truncated once the 100 MB boundary is reached, potentially retaining only a few hours or days of history.

External Log Forwarding (Splunk, Datadog, ELK)

To retain logs indefinitely, comply with regulatory audit requirements, or analyze logs centrally across an enterprise, organizations configure external log forwarders:

1. Custom log4j2.xml Appenders: Developers modify the project's log4j2.xml configuration to include direct TCP, UDP, or HTTP appenders targeting external logging platforms (e.g., Splunk HTTP Event Collector, Datadog API, Logstash).
2. Disabling CloudHub Default Appender: When deploying a custom log4j2.xml appender, developers must request MuleSoft Support or configure the application property log4j2.enable.default.appender=false to prevent duplicate logging and reduce worker I/O overhead.
3. Anypoint Titanium: An enterprise add-on that provides built-in, out-of-the-box log archiving, centralized search across all applications, and extended retention up to 200 GB per vCore.

---

6. Anypoint Monitoring & Telemetry Analytics

Anypoint Monitoring provides comprehensive observability into application health, performance bottlenecks, and resource utilization.

+-----------------------------------------------------------------------------------------+
|                             ANYPOINT MONITORING DASHBOARDS                              |
|                                                                                         |
|   +--------------------------+  +--------------------------+  +---------------------+   |
|   | INBOUND RESPONSE TIME    |  | THROUGHPUT (RPM)         |  | ERROR RATE (HTTP 5xx|   |
|   | Avg: 42ms  99th: 180ms   |  | Current: 4,200 req/min   |  | Failures: 0.02%     |   |
|   +--------------------------+  +--------------------------+  +---------------------+   |
|                                                                                         |
|   +--------------------------+  +--------------------------+  +---------------------+   |
|   | JVM HEAP UTILIZATION     |  | THREAD POOL METRICS      |  | FLOW-LEVEL BREAKDOWN|   |
|   | Used: 420 MB / 1024 MB   |  | Active: 18  Queued: 0    |  | /orders: 22ms       |   |
|   +--------------------------+  +--------------------------+  +---------------------+   |
+-----------------------------------------------------------------------------------------+

Key Observability Dimensions:

• Inbound & Outbound Latency: Tracks average, 50th, 90th, and 99th percentile response times for all HTTP endpoints and external connector calls.
• Message Throughput & Error Counts: Graphs requests per minute (RPM) segmented by HTTP response status codes (2xx, 4xx, 5xx).
• JVM Health Metrics: Real-time visibility into JVM heap vs. non-heap memory, garbage collection (GC) pause durations, and thread pool saturation.
• Flow-Level Diagnostics: Isolates latency down to individual message processors, revealing slow database queries or lagging third-party HTTP endpoints.

---

7. Exam Watch: Core Runtime Management Scenarios

[!IMPORTANT] Triggering Custom Alerts On the Developer I exam, if a question asks how to trigger an email notification when a specific business condition or connector error occurs inside a flow, the answer is to use the CloudHub Connector's <cloudhub:create-notification> operation in conjunction with a Runtime Manager custom alert.

[!WARNING] The 100 MB / 30-Day Log Limit Always remember that CloudHub log retention is 100 MB or 30 days, whichever comes first. For high-volume production applications, logs will be deleted long before 30 days unless forwarded to an external log tool like Splunk or Datadog.

[!TIP] Rolling Update Guarantees A CloudHub rolling update never drops active transactions. The old worker is only terminated after the new worker passes health checks and active requests on the old worker finish processing.