12.3 Disaster Recovery, Redundant Connectivity & Evacuation

12.3 Disaster Recovery, Redundant Connectivity & Evacuation

Quick Answer: Disaster Recovery (DR) and Continuity of Operations (COOP) are complementary public safety disciplines: COOP addresses people, processes, leadership succession, and operational mission survival (PMEFs), while DR focuses on technical restoration of IT systems, databases, network connectivity, and hardware. Key recovery metrics include Recovery Time Objective (RTO) (maximum allowable downtime) and Recovery Point Objective (RPO) (maximum acceptable data loss in time). Mission-critical CAD architectures deploy synchronous database replication locally (RPO = 0) and asynchronous replication across geographically separated wide-area links. Public safety data is secured against ransomware via the 3-2-1 immutable backup rule. Telecommunications ingress must enforce true physical diversity (diverse conduits, separate building vaults, distinct carrier backbones, and BGP-4 multihoming). During sudden facility emergencies, immediate life safety is the absolute first priority, supported by automated 9-1-1 call diversion, tactical Go-Kits, and mobile Tactical Dispatch Teams (TDTs).

---

1. Disaster Recovery (DR) vs. Continuity of Operations (COOP)

Emergency Number Professionals must understand the operational boundary and functional intersection between COOP and Disaster Recovery:

┌────────────────────────────────────────┬────────────────────────────────────────┐
│ CONTINUITY OF OPERATIONS (COOP)        │ DISASTER RECOVERY (DR)                 │
├────────────────────────────────────────┼────────────────────────────────────────┤
│ • People, leadership, and operational  │ • Technology, servers, databases, and  │
│   workflow focused.                    │   telecommunications networks focused. │
│ • Preserves Primary Mission Essential  │ • Restores technical computing platforms│
│   Functions (PMEFs).                   │   and data repositories to service.    │
│ • Manages evacuation logistics, staff  │ • Manages database rebuilds, server    │
│   transit, and alternate site seating. │   failover, SAN replication, and WANs. │
│ • Driven by operational supervisors,  │ • Driven by systems engineers, network │
│   ECC directors, and public safety SOPs│   architects, and hardware vendors.    │
└────────────────────────────────────────┴────────────────────────────────────────┘

While COOP dictates where telecommunicators will sit and how they will dispatch calls, Disaster Recovery provides the replicated CAD data, restored telephony paths, and synchronized radio networks that allow those telecommunicators to function.

---

2. Mission-Critical Recovery Metrics: RTO, RPO & MTTR

In public safety system architecture, vague recovery goals such as "restore as fast as possible" are unacceptable. System resilience is governed by mathematically bounded metrics established through a formal Business Impact Analysis (BIA):

[Last Data Commit / Backup]             [Disaster Occurs]            [System Restored]
            │                                   │                            │
            ├───────────────────────────────────┼────────────────────────────┤
            │◄──────── RPO Window ─────────────►│◄─────── RTO Window ───────►│
            │ (Max acceptable data loss in time)│ (Max acceptable downtime)  │

• Recovery Time Objective (RTO): The maximum tolerable length of time that a system, network, or application can remain offline following a catastrophic event before severe operational failure occurs. RTO looks forward in time from the moment of disruption to the moment of restoration.
• Recovery Point Objective (RPO): The maximum acceptable age of transactional data that can be permanently lost when a catastrophic event strikes, measured backward in time from the failure to the most recent uncorrupted data commit. RPO determines the required replication frequency and database architecture.
• Mean Time Between Failures (MTBF): The statistical predicted elapsed operational time between inherent system failures during normal operations.
• Mean Time to Repair (MTTR): The average time required to troubleshoot, replace, and restore a failed hardware or software component to service.

Public Safety System Benchmarks

---

3. Database Resiliency: Synchronous vs. Asynchronous Replication

To prevent losing active incident cards, unit statuses, and call logs during a server room catastrophe, CAD databases cannot rely solely on periodic backups. They require real-time database replication.

SYNCHRONOUS REPLICATION (Zero Data Loss - Distance Limited)
[CAD App] ──► [Primary DB] ──► (Network Write) ──► [Secondary DB]
                 ▲                                      │
                 │◄────────── (Write Confirmed) ────────┘
[Transaction Confirmed to CAD]
* Guarantees RPO = 0. Round-trip network latency limits distance (< 50–100 km).

ASYNCHRONOUS REPLICATION (Zero Latency Penalty - Unlimited Distance)
[CAD App] ──► [Primary DB] ──► [Transaction Confirmed to CAD Immediately]
                    │
                    └──► (Pushed across WAN in background) ──► [Remote DR DB]
* Eliminates application lag; allows 500+ mile separation; minimal RPO risk (seconds).

Synchronous Replication

In synchronous database replication, when the CAD application writes a record (e.g., dispatching an EMS unit), the primary database writes the transaction locally and simultaneously transmits it to a secondary database node. The primary database does not commit the transaction or confirm completion to the CAD application until the secondary node acknowledges that the data has been successfully written.

• Advantage: Guarantees absolute zero data loss (RPO = 0). If the primary server suddenly loses power, the secondary replica is identical.
• Constraint: The speed of light in fiber optic cables (~5 microseconds per kilometer) and network switching introduce round-trip latency. If the secondary data center is 300 miles away, every synchronous write waits for a round trip (several milliseconds of fiber delay plus network and database processing), which can make CAD noticeably sluggish. For that reason, synchronous replication is usually kept to local or metro data centers (commonly within about 100 kilometers).

Asynchronous Replication

In asynchronous replication, the primary database commits the transaction locally and confirms completion to the CAD application immediately. In the background (milliseconds later), the transaction log is batched and transmitted across the Wide Area Network (WAN) to the remote disaster recovery database.

• Advantage: Zero application latency penalty. CAD operates at full speed, and the remote database can be located hundreds of miles away in another state, completely outside the regional disaster footprint.
• Constraint: Introduces a fractional-second "replication lag." If a catastrophe destroys the primary site before data in flight reaches the remote node, transactions in that micro-window are lost, yielding an RPO of seconds.

The Hybrid Public Safety Architecture

Leading public safety centers implement a three-node hybrid model: A primary CAD server in Data Center 1 synchronously replicates to an active-passive node in Data Center 2 (located 25 miles away, delivering RPO=0 high availability), while asynchronously replicating to Data Center 3 located 400 miles away in an independent power grid (providing catastrophic disaster recovery).

---

4. Immutable Backups & The 3-2-1 Rule

Ransomware represents one of the greatest operational threats to public safety IT infrastructure. Modern threat actors deliberately compromise administrative domain credentials, locate backup servers, and delete or encrypt all backup volumes before encrypting production databases.

                    THE 3-2-1 IMMUTABLE BACKUP RULE
┌─────────────────────────────────────────────────────────────┐
│ 3 Total Copies of Critical Data                             │
│ (1 Primary Production Volume + 2 Dedicated Backups)         │
├─────────────────────────────────────────────────────────────┤
│ 2 Different Storage Media Formats                           │
│ (e.g., High-Speed Local NVMe/SAN Array + Cloud Object Store)│
├─────────────────────────────────────────────────────────────┤
│ 1 Copy Stored Off-Site in an Immutable / WORM Repository    │
│ (Write-Once-Read-Many; Cryptographically Locked Against     │
│  Deletion or Modification Even by Domain Administrators)    │
└─────────────────────────────────────────────────────────────┘

Write-Once-Read-Many (WORM) & Object Locking

To ensure survivability against ransomware, off-site backups must enforce Immutability (WORM storage):

• Once written, backup objects cannot be overwritten, encrypted, renamed, or deleted by any user, process, API call, or compromised domain administrator account until a predetermined legal retention timer (e.g., 30 or 90 days) expires.
• Cloud object storage repositories (such as AWS S3 Glacier Object Lock in Compliance Mode or Azure Immutable Blob Storage) enforce kernel-level cryptographic immutability that cannot be overridden even by cloud account administrators.

---

5. Redundant Telecommunications Ingress & Physical Path Diversity

A resilient public safety communications center cannot rely on a single fiber connection, regardless of promised carrier reliability. True network path diversity requires eliminating every single physical point of failure between the serving central offices and the dispatch floor.

                             [Public Safety ECC Facility]
                                   │               │
         ┌─────────────────────────┘               └─────────────────────────┐
         ▼                                                                   ▼
[Entrance Vault A (North Wall)]                                   [Entrance Vault B (South Wall)]
         │ (Conduit Path 1 - North Street)                                   │ (Conduit Path 2 - South Ave)
         ▼                                                                   ▼
[Carrier A - Fiber Optic Ring]                                    [Carrier B - Fiber Optic Ring]
         │                                                                   │
         ▼                                                                   ▼
[Central Office 1 (Downtown)]                                     [Central Office 2 (Uptown)]

Four Layers of Network Diversity:

1. Diverse Physical Conduits: External fiber cables must travel through completely separate underground trenching routes and duct banks along separate municipal street rights-of-way. Minimum physical separation (typically at least 50 to 100 feet) ensures that a single construction excavator cannot sever both paths.
2. Diverse Building Entrances: Feeder conduits must penetrate the building through two separate entrance vaults (Points of Presence - POPs) located on opposite exterior walls of the structure (e.g., North entrance and South entrance).
3. Carrier & Central Office Diversity: Circuits must be contracted through two completely independent Tier 1 telecommunications service providers utilizing separate fiber sheaths and routing to two distinct carrier Central Office wire centers.
4. Internal Cable Risers: Inside the facility, cables from Entrance Vault A and Entrance Vault B must route through separate, fire-rated risers and conduit runs to redundant telecommunications equipment rooms.

BGP Multihoming & Autonomous System Routing

At the network routing layer, redundant internet and ESInet connections must deploy Border Gateway Protocol version 4 (BGP-4) Multihoming:

• The 9-1-1 authority obtains its own public Autonomous System Number (ASN) and Provider-Independent (PI) IP address space from the regional internet registry (ARIN).
• Edge routers establish BGP peering sessions with both independent service providers simultaneously.
• If Carrier A suffers a fiber break or core routing failure, BGP routing tables automatically withdraw Carrier A's routes and redirect all incoming and outgoing IP traffic across Carrier B in seconds, with zero renumbering of internal CAD servers, IP phones, or dispatch consoles.

---

6. PSAP Evacuation Protocols & Tactical Execution

When a catastrophic facility event (such as a structural breach, chemical spill, or fire) requires abandoning the dispatch floor, the evacuation must execute under structured, pre-planned standard operating procedures.

Life Safety as the Paramount Priority

In any sudden, life-threatening emergency, the immediate life safety of communications personnel is the first and absolute priority. Telecommunicators must never delay evacuation to dismount server hardware, pack paper files, or attempt manual data backups. Facilities and electronics are replaceable; human lives are not.

Tactical Evacuation Sequence:

1. Fast Emergency Transfer Initiation (Immediate - < 30 Seconds): Before stepping away from the consoles, the shift supervisor activates the automated 9-1-1 call diversion macro. In NG911 environments, this triggers an administrative PRF policy routing rule at the ESRP, redirecting all incoming calls to the backup center or mutual aid partner. In legacy environments, the supervisor hits a speed-dial macro or alerts the carrier NOC.
2. Immediate Field Responder Notification: Broadcast a standardized emergency evacuation announcement over all primary dispatch talkgroups (e.g., "All units, Communications is evacuating the primary facility due to an emergency. Mutual aid partner County Central is assuming primary dispatch authority until further notice.").
3. Deployment of the Tactical Go-Kit: The designated supervisor grabs the hardened Go-Kit, which contains pre-configured portable radios, extra batteries, cellular hotspots, MFA hardware tokens, administrative credential rosters, and facility master fobs.
4. Evacuation & Personnel Accountability: Staff exit via designated emergency routes to the primary assembly point. A formal Personnel Accountability Report (PAR) is completed immediately.
5. Deployment of Tactical Dispatch Teams (TDT): A dedicated, pre-designated cadre of tactical dispatchers transits directly to the alternate backup facility or on-scene Tactical Command Post equipped with portable radio caches to re-establish primary dispatch operations.

---

7. Post-Incident Debriefing: Hotwash, AAR & Improvement Plans

The disaster recovery lifecycle concludes with translating operational challenges into institutional policy improvements following the Homeland Security Exercise and Evaluation Program (HSEEP) model:

[Disaster Recovery / Evacuation Event Concludes]
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│ 1. Operational Hotwash (Immediate - within 2 to 4 hours)    │
│    - Informal, raw capture of operational friction points   │
├─────────────────────────────────────────────────────────────┤
│ 2. Multi-Agency Debriefing (Within 5 to 7 business days)    │
│    - Structured timeline analysis across all stakeholders   │
├─────────────────────────────────────────────────────────────┤
│ 3. After Action Report (AAR) (Formal written document)       │
│    - Detailed chronological analysis of operational actions │
├─────────────────────────────────────────────────────────────┤
│ 4. Improvement Plan (IP) (Actionable accountability matrix) │
│    - Specific corrective tasks, assigned owners, and        │
│      measurable completion deadlines                        │
└─────────────────────────────────────────────────────────────┘

The Improvement Plan (IP) Matrix

An After Action Report without an actionable Improvement Plan (IP) is useless. The IP must convert findings into measurable accountability tasks:

---

8. Operational Traps & ENP Exam Watch

• COOP vs. Disaster Recovery: COOP is operational and organizational (people, leadership, mission functions, alternate sites). DR is technical and data-centric (restoring servers, databases, and network connectivity).
• Synchronous Replication Over Long Distances: A common engineering mistake is attempting synchronous database replication across facilities separated by hundreds of miles. Round-trip network latency will freeze active CAD consoles. Long-distance replication must always be asynchronous.
• True Network Diversity vs. Shared Conduits: Two fiber contracts from different vendors do not provide diversity if both carriers share the same underground conduit, bridge crossing, or utility trench. True diversity requires separate physical pathways from the street to the server room.
• Life Safety Supersedes Equipment: In any sudden emergency evacuation scenario, never choose an answer option that involves delaying evacuation to dismount servers, back up files, or salvage equipment.