11.4 IoT Incident Handling and Firmware Investigation

IoT Incident Handling and Firmware Investigation

Internet of Things (IoT) incidents rarely live on one device. A camera, sensor, badge reader, medical peripheral, or building controller may depend on a local gateway, mobile companion application, message broker, vendor cloud, identity service, and fleet-management console. The incident handler must reconstruct this system of systems while recognizing that the endpoint may have little storage, no reliable clock, and a physical function that cannot be interrupted casually.

Build the Evidence Map

Start with an inventory tied to observed identifiers rather than a product label alone:

• manufacturer, model, hardware revision, serial number, asset tag, MAC addresses, radio identifiers, and installed firmware version;
• IP address and DHCP history, switch port, wireless association, VLAN or segment, gateway, DNS and NTP configuration;
• device certificate, client ID, service account, broker topics, API tokens, and fleet or tenant identifiers without exposing secret values in general tickets;
• management console, vendor cloud region, mobile application, update service, local hub, and accounts authorized to administer the device;
• physical process or business function, approved maintenance window, owner, vendor support path, and safety consequence of disconnection.

Record time sources carefully. Low-cost devices often reset clocks after reboot, drift substantially, or log local time without a zone. Correlate device records with DHCP, wireless controllers, gateways, broker logs, packet captures, cloud audit events, and physical observations.

Detection and Triage

Common signals include outbound scanning, repeated DNS lookups, unexpected broker connections, new administrator accounts, configuration changes, disabled updates, abnormal power or sensor behavior, and traffic to known command infrastructure. A device sending high-volume UDP traffic may be a botnet node, but first rule out a legitimate firmware update, telemetry burst, or misconfigured discovery protocol.

Scope the fleet. Shared factory credentials, cloned certificates, a vulnerable update package, or one exposed management API can turn a single alert into a product-wide incident. Query management platforms for devices with the same model, firmware, certificate issuer, configuration template, or external destination.

Safety-Aware Containment

Containment can occur at several layers:

Coordinate with the device owner, facilities or clinical engineering where applicable, safety personnel, vendor, and legal team. If the device controls a physical process, containment authority belongs in the approved operational playbook—not solely with the SOC.

Firmware Acquisition

Prefer the least invasive source that answers the investigative question. Possible sources include a vendor-published signed firmware image, update server cache, management-console export, device backup, mounted file system, bootloader or UART console, JTAG interface, and a direct SPI or eMMC read. Hardware methods can alter state, trip tamper controls, void support, or damage components; use trained personnel, photograph connections, preserve the original chip when feasible, and hash each acquired image.

Tools such as binwalk can identify embedded file-system and compression signatures and extract supported formats such as SquashFS. Extraction is not proof of compromise. Analysts inspect startup scripts, web interfaces, update verification, hardcoded credentials, certificates, scheduled tasks, kernel modules, and differences from a known vendor image. Architecture matters: ARM, MIPS, and other binaries require the correct disassembler or controlled emulation environment.

Protocol and Backend Analysis

MQTT commonly uses a broker and hierarchical topics; CoAP exposes resources over UDP and can use Datagram Transport Layer Security (DTLS) or other protection profiles. Neither protocol is inherently an incident. Investigate:

• anonymous broker access, broad wildcard topic permissions, retained malicious messages, unexpected publisher client IDs, and credentials reused across devices;
• cleartext transport where sensitive data or commands require protection, weak certificate validation, exposed management ports, and unauthenticated firmware endpoints;
• command provenance: which identity published the instruction, from what address, through which gateway, and whether the device acknowledged it;
• backend compromise, because a validly signed command from a stolen cloud administrator session may look normal to the endpoint.

Eradication and Recovery

Preserve evidence before reimaging. Replace unauthorized firmware with a vendor-validated image, verify digital signatures and secure-boot state where supported, remove rogue accounts and startup entries, rotate unique device certificates and backend tokens, patch the gateway and management plane, and eliminate default or shared credentials. Re-enroll devices in stages, confirm expected telemetry and physical behavior, and watch for repeated outbound destinations or configuration rollback.

The final report separates confirmed device compromise, backend account compromise, vulnerable configuration, and suspected but unverified behavior. Fleet-wide corrective actions—certificate rotation, broker authorization, signed updates, segmentation, asset inventory, and end-of-support replacement—often matter more than cleaning one endpoint.