16.1 Cisco Embedded Event Manager (EEM) Applet Architecture and Event Detectors

Cisco Embedded Event Manager (EEM) Applet Architecture and Event Detectors

Enterprise network operations demand rapid, deterministic responses to transient link faults, unauthorized administrative changes, and telemetry threshold violations. While centralized Network Management Systems (NMS) and telemetry collectors provide global operational visibility, the latency involved in detecting an outage, alerting an engineer, and dispatching remediation commands over in-band or out-of-band management networks can span seconds or minutes. Cisco Embedded Event Manager (EEM) resolves this operational challenge by embedding automated event detection and programmatic remediation directly inside the network operating system (Cisco IOS and Cisco IOS-XE). Operating locally on the switch or router control plane, EEM executes corrective actions within milliseconds to seconds without waiting for an external controller.

+-------------------------------------------------------------------------+
|                    Cisco EEM Three-Tier Architecture                    |
+-------------------------------------------------------------------------+
|  EVENT DETECTORS (Subsystem Monitoring)                                 |
|  - Syslog Detector    - CLI Detector      - Track Detector              |
|  - Timer Detector     - Interface Counter - SNMP / IP SLA               |
+-------------------------------------------------------------------------+
                                     |
                                     | Event Notification & Context Metadata
                                     v
+-------------------------------------------------------------------------+
|  EEM SERVER (Core Orchestration Engine)                                 |
|  - Matches Trigger to Registered Policies                               |
|  - Manages Execution Priorities & Concurrency Locks                     |
|  - Populates Runtime Environment Variables ($_syslog_msg, etc.)         |
+-------------------------------------------------------------------------+
                                     |
                                     | Invocation Trigger
                                     v
+-------------------------------------------------------------------------+
|  EVENT POLICIES (Subscribers / Remediation Actions)                     |
|  +---------------------------------+   +-----------------------------+  |
|  |     EEM Applets (CLI Based)     |   |    EEM Tcl Scripts (.tcl)   |  |
|  | - Defined in global config mode |   | - Complex logic & loops     |  |
|  | - Alphanumeric action labels    |   | - Network sockets & file IO |  |
|  +---------------------------------+   +-----------------------------+  |
+-------------------------------------------------------------------------+

EEM Architecture: Three-Tier Functional Model

EEM operates through a decoupled, event-driven publisher-subscriber architecture consisting of three primary functional elements:

1. Event Detectors (Publishers)

Event Detectors are specialized, low-level software subsystems embedded within Cisco IOS and IOS-XE that continuously monitor specific operational domains. Rather than requiring continuous CPU polling, Event Detectors hook directly into internal software buses, interrupt routines, and daemon messaging queues. Monitored conditions include incoming syslog messages, user CLI commands, Enhanced Object Tracking (EOT) state transitions, hardware interface packet counters, and timer expirations. When a monitored condition satisfies the detector's configured criteria, the Event Detector packages the event metadata and publishes a notification to the EEM Server.

2. EEM Server (Mediation Engine)

The EEM Server serves as the central orchestration daemon operating within the control plane. It receives published notifications from Event Detectors, references the internal policy registry to locate matching event policies, evaluates execution priority and concurrency rules, populates runtime environment variables with event context, and invokes the registered policy.

3. Event Policies (Subscribers)

Event Policies define the automated workflows triggered when an event occurs. EEM supports two distinct implementation mechanisms:

• EEM Applets: Simple, declarative policies defined directly within global configuration mode using standard Cisco IOS CLI syntax (event manager applet <name>). Applets are straightforward to implement, require no external programming runtime, and handle common operational tasks like running sequential CLI commands, logging messages, and sending alerts.
• EEM Tcl Scripts: Advanced, procedural scripts written in Tool Command Language (Tcl) and stored on local file systems (such as bootflash:). Tcl scripts support complex computational logic, nested conditional branches, while/for loops, network socket communication, external file I/O, and custom cryptographic operations.

Core Event Detectors

EEM provides a robust suite of event detectors tailored to enterprise infrastructure monitoring:

Syslog Event Detector (event syslog)

The syslog detector monitors the internal logging process. When an incoming log message matches a specified regular expression pattern, the detector fires:

event manager applet MONITOR-OSPF-NEIGHBOR
 event syslog pattern "%OSPF-5-ADJCHG: Process 1, Nbr .* on GigabitEthernet0/1 from FULL to DOWN"

Modifiers include occurs <num> (requires the pattern to appear a set number of times before triggering) and period <seconds> (defines the sliding time window in which the pattern occurrences must take place).

Timer Event Detector (event timer)

The timer detector enables time-driven scheduling across three operational modes:

• Cron (cron): Schedules policy execution according to standard calendar and time specifications (e.g., event timer cron name DAILY-BACKUP cron-entry "0 2 * * *" triggers daily at 2:00 AM).
• Countdown (countdown): Triggers once after a designated duration following system boot or policy registration (e.g., waiting 120 seconds after bootup before verifying BGP peering).
• Watchdog (watchdog): Operates as a recurring periodic interval timer (e.g., event timer watchdog time 300 triggers every 300 seconds continuously).

CLI Event Detector (event cli)

The CLI detector intercepts commands entered by administrative users or automated scripts at user or privileged EXEC prompts:

event manager applet PREVENT-RELOAD
 event cli pattern "reload" sync yes

The sync parameter determines execution timing:

• sync yes (Synchronous): The CLI parser suspends execution of the user's command and hands control to the EEM applet. The applet inspects the command and selectively allows or prohibits execution using the built-in variable _exit_status. Setting action 99 set _exit_status 0 blocks the command from executing and returns an error to the user terminal, whereas setting _exit_status 1 permits the command.
• sync no (Asynchronous): The command executes immediately in the terminal, while the EEM applet runs concurrently in the background.

Enhanced Object Tracking Event Detector (event track)

The track detector hooks into Cisco Enhanced Object Tracking (EOT). EOT can track IP SLA probe reachability, route presence in the Routing Information Base (RIB), or interface line protocol states:

event manager applet WAN-FAILOVER
 event track 10 state down

When the tracked object transitions to the specified state (up, down, or any), EEM initiates immediate convergence actions.

Interface Event Detector (event interface)

The interface detector monitors low-level physical interface statistics, such as input errors (input_errors), CRC errors (input_errors_crc), output errors (output_errors), or receive rates. The entry-type keyword says whether the threshold applies to the counter value, its increase since the last poll (increment), or its rate:

event manager applet CHECK-PORT-ERRORS
 event interface name GigabitEthernet0/2 parameter input_errors entry-op ge entry-val 100 entry-type increment poll-interval 30

When errors exceed the configured threshold within the polling interval, EEM acts proactively before complete physical link failure occurs.

Event Detectors Comparison Matrix

Action Statements and Execution Ordering

Action statements define the programmatic steps executed by an EEM applet. Each action begins with the keyword action, followed by an alphanumeric label, and the specific functional command:

action <label> <command> [arguments]

Critical Principle: Alphanumeric (Lexicographical) Ordering

EEM executes action statements in strict alphanumeric (ASCII) label order, not in the order they were entered in configuration mode. If labels are numbered 1, 2, and 10, the ASCII sort sequence executes action 1, followed by action 10, and then action 2. If action 10 depended on action 2 executing first, the applet fails.

To ensure deterministic sequential execution, engineers must use padded numeric labels such as 10, 20, 30, zero-padded strings like 01, 02, 03, or decimal notation like 1.0, 2.0, 3.0.

Action Statement Syntax Reference

Built-in Environment Variables

When an event fires, the EEM Server automatically populates built-in context variables prefixed with $_. These variables pass event metadata directly into the applet's execution context:

• $_syslog_msg: Contains the exact string of the syslog message that triggered an event syslog detector.
• $_cli_result: Stores the output of the most recent action cli command in the applet. In synchronous CLI applets, setting _exit_status to 0 blocks execution, while 1 allows it.
• $_cli_msg: The full command string typed by the administrative user in an event cli applet.
• $_event_pub_time: The timestamp indicating exactly when the event detector published the event to the EEM Server.
• $_track_state: The new state (up or down) of a tracked object that triggered an event track applet.

Practical EEM Applet Walkthroughs

Scenario 1: Unauthorized Configuration Command Interception and Prevention

In high-security enterprise environments, destructive commands such as write erase must be blocked, logged, and alerted in real time:

event manager applet BLOCK-WRITE-ERASE
 description "Intercepts and blocks write erase commands synchronously"
 event cli pattern "write erase" sync yes
 action 10 syslog priority critical msg "SECURITY ALERT: Unauthorized 'write erase' attempted by user!"
 action 20 snmp-trap strdata "CRITICAL: 'write erase' command blocked on core router"
 action 30 set _exit_status "0"

In this applet, sync yes intercepts the command before the parser executes it. action 10 logs a critical syslog message, action 20 sends an SNMP trap, and action 30 sets _exit_status 0, which discards the command and returns an error to the administrator's terminal.

Scenario 2: Automated Interface Remediation upon Error Detection

When an access interface experiences physical errors, bouncing the port can clear transient transceiver register faults:

event manager applet BOUNCE-FLAPPING-PORT
 description "Bounces GigabitEthernet0/1 when input errors exceed threshold"
 event interface name GigabitEthernet0/1 parameter input_errors entry-op ge entry-val 50 entry-type increment poll-interval 15
 action 10 syslog priority warning msg "EEM: Gi0/1 input errors rose by 50 or more. Bouncing the port."
 action 20 cli command "enable"
 action 30 cli command "configure terminal"
 action 40 cli command "interface GigabitEthernet0/1"
 action 50 cli command "shutdown"
 action 60 wait 5
 action 70 cli command "no shutdown"
 action 80 cli command "end"
 action 90 syslog priority informational msg "EEM: Port Gi0/1 successfully bounced."

Scenario 3: Dynamic Route Flap Troubleshooting and Diagnostic Capture

Intermittent BGP route flaps can disappear before engineers can troubleshoot. EEM can trigger automated diagnostic capture upon neighbor loss:

event manager applet CAPTURE-BGP-FLAP
 description "Captures routing table and traceroute upon BGP neighbor loss"
 event syslog pattern "%BGP-5-ADJCHANGE: neighbor 192.0.2.1 Down"
 action 10 syslog priority alert msg "EEM: BGP Neighbor 192.0.2.1 Down! Collecting diagnostics."
 action 20 cli command "enable"
 action 30 cli command "show ip route 192.0.2.1 | append flash:bgp_debug.txt"
 action 40 cli command "show ip bgp neighbors 192.0.2.1 | append flash:bgp_debug.txt"
 action 50 cli command "show ip bgp summary | append flash:bgp_debug.txt"
 action 60 syslog priority informational msg "EEM: Diagnostics appended to flash:bgp_debug.txt"

Verification and Troubleshooting Commands

Verifying EEM operational status and execution history requires several privileged EXEC commands:

• show event manager policy registered: Displays all registered EEM applets and Tcl scripts, showing their trigger conditions, class, and event detector attributes.
• show event manager history events: Displays the execution log of recent EEM events, including timestamps, event detector types, and triggered policy names.
• show event manager environment: Lists all configured EEM environment variables and built-in system variables.
• debug event manager action cli: Provides real-time console tracing of virtual CLI commands executed within active EEM applets, essential for troubleshooting command syntax or authentication errors.