17.4 Emergency Response Planning & Plant Physical/Cyber Security
Key Takeaways
- Section 2013 of America's Water Infrastructure Act (AWIA) of 2018 requires all Community Water Systems serving over 3,300 persons to develop a Risk and Resilience Assessment (RRA) and Emergency Response Plan (ERP), recertifying compliance to the EPA every 5 years.
- The Incident Command System (ICS), structured under the National Incident Management System (NIMS), organizes disaster response into five core functions: Command, Operations, Planning, Logistics, and Finance/Administration.
- Physical plant security relies on a defense-in-depth framework incorporating perimeter intrusion fencing, electronic access logging, locked chemical containment, and intrusion-alarmed, gasketed finished water storage hatches with 24-mesh vent screens.
- Industrial control and SCADA cybersecurity requires strict network segmentation between Information Technology (IT) and Operational Technology (OT) networks, Multi-Factor Authentication (MFA) for remote access, elimination of default passwords, and immutable offline backups.
- Emergency water supply planning requires maintaining a minimum emergency allocation of 1 to 2 gallons per person per day, mutual aid via WARN agreements, and issuing Boil Water Advisories requiring a 1-minute rolling boil whenever distribution pressure drops below 20 psi.
America's Water Infrastructure Act of 2018 (AWIA Section 2013)
Drinking water treatment and distribution facilities represent critical lifeline infrastructure whose disruption can cripple municipal fire protection, industrial manufacturing, healthcare facilities, and public sanitation. In 2018, Congress passed America's Water Infrastructure Act (AWIA), enacting Section 2013 (which amended Section 1433 of the SDWA, permanently superseding the older Public Health Security and Bioterrorism Preparedness and Response Act of 2002).
Statutory Requirements and Applicability
AWIA mandates that every Community Water System (CWS) serving a population greater than 3,300 individuals must prepare two comprehensive emergency planning instruments:
- Risk and Resilience Assessment (RRA): A comprehensive evaluation of the system's vulnerability to potential malevolent acts and natural hazards.
- Emergency Response Plan (ERP): An actionable operational playbook detailing how the utility will prepare for, respond to, mitigate, and recover from malevolent acts or natural disasters.
The Mandatory 5-Year Recertification Cycle
A core component of AWIA is that water utilities must not allow security planning to become static shelfware. Under the statute, every CWS serving > 3,300 must review, update if necessary, and recertify both its RRA and ERP to the EPA at least once every 5 years. Following the initial statutory compliance phase (2020–2021), utilities must recertify on rolling 5-year cycles (e.g., 2025–2026, 2030–2031). Rather than submitting the sensitive security documents themselves, utilities submit an official electronic certification letter signed by an authorized municipal official directly into the secure EPA portal.
Required Scope of the Risk and Resilience Assessment (RRA)
Under AWIA, the RRA must evaluate the resilience of:
- Physical Infrastructure: Vulnerability of raw water intakes, chemical storage facilities, filtration galleries, finished water clearwells, booster pump stations, and distribution trunk mains.
- Threat Vectors: Malevolent human acts (terrorism, cyber warfare, vandalism, active shooters, hazardous material contamination) and natural hazards (floods, hurricanes, tornadoes, earthquakes, droughts, winter freezes).
- Cybersecurity Architecture: Vulnerabilities within Supervisory Control and Data Acquisition (SCADA) systems, Programmable Logic Controllers (PLCs), business IT networks, and remote cellular telemetry.
- Chemical Storage Hazards: Catastrophic release models for hazardous gaseous chemicals, specifically 150-lb cylinders and 1-ton containers of liquefied chlorine gas and anhydrous ammonia.
- Supply Chain and Financial Systems: Single-source chemical supply vulnerabilities, power grid failure contingencies, and critical mechanical spare parts availability.
Required Scope of the Emergency Response Plan (ERP)
Within six months of completing and certifying the RRA, the utility must certify its updated ERP, which must incorporate:
- Specific strategies and automated procedures to improve overall utility resilience against identified malevolent and natural hazards.
- Comprehensive plans, operational protocols, and mutual aid procedures for responding immediately to emergency incidents.
- Security equipment maintenance and testing schedules (intrusion alarms, backup generators, cybersecurity firewalls).
- Alternate drinking water supply strategies, including emergency interconnections, bulk water hauling, and bottled water staging.
Incident Command System (ICS) and NIMS Framework
During large-scale water emergencies (such as main transmission breaks, chemical spills, power blackouts, or contamination events), water utility personnel must coordinate seamlessly with local fire departments, hazardous materials teams, emergency medical services, law enforcement, and federal agencies. Under Homeland Security Presidential Directive 5 (HSPD-5), municipal utilities are mandated to operate within the National Incident Management System (NIMS) using the Incident Command System (ICS).
Core Functional Management Areas of ICS
The ICS organizational structure operates under five core management functions:
- Incident Command (Incident Commander): Holds ultimate responsibility for overall incident management, life safety, scene control, establishing incident objectives, and approving the Incident Action Plan (IAP). Operates with a manageable span of control (typically 3 to 7 individuals, with 5 being optimal).
- Operations Section: Directs and executes all tactical operations to carry out the Incident Action Plan. In a water utility context, Operations personnel perform line isolations, operate pumps, deploy emergency bypass piping, contain chemical leaks, and manage chemical dosage adjustments.
- Planning Section: Responsible for collecting, evaluating, analyzing, and tracking operational information regarding incident status and resource deployment. The Planning Section tracks resource status, assesses documentation, forecasts incident progression, and authors the written IAP for subsequent operational periods.
- Logistics Section: Procures and provides all necessary resources, facilities, services, and materials needed to sustain incident responders. In a water emergency, Logistics coordinates emergency diesel fuel delivery for standby generators, procures repair clamps and vacuum trucks, provides portable restrooms and food, and establishes emergency radio communications channels.
- Finance / Administration Section: Tracks all financial expenditures, vendor invoices, personnel overtime hours, and worker injury claims associated with the incident. Crucially, the Finance Section maintains meticulous documentation required for municipal financial audits and federal disaster reimbursement grants (FEMA Public Assistance).
Command Staff Functions
The Incident Commander is directly supported by three specialized Command Staff officers:
- Safety Officer: Assesses hazardous conditions and develops safety measures for response personnel. Crucially, the Safety Officer possesses the explicit, autonomous authority to bypass the chain of command and immediately halt or suspend any tactical operation deemed an imminent danger to life or health.
- Public Information Officer (PIO): Serves as the single designated conduit between the Incident Command and external news media, civic officials, and the public. The PIO prepares press releases, coordinates media briefings, and issues emergency public notices (preventing conflicting messages).
- Liaison Officer: Acts as the primary point of contact for assisting and cooperating external agencies, including representatives from the EPA, state primacy agency, public health department, electric power utilities, and regional mutual aid networks.
[ Incident Command System (ICS) Water Emergency Organizational Hierarchy ]
+----------------------------+
| INCIDENT COMMANDER |
+--------------+-------------+
|
+----------------------------+----------------------------+
| | |
+-------+--------+ +--------+-------+ +---------+------+
| SAFETY OFFICER | | PUBLIC INFO | | LIAISON |
| (Halts Unsafe | | OFFICER (PIO) | | OFFICER |
| Tactics) | +----------------+ +----------------+
+----------------+
|
+------------------+---------------+------------------+------------------+
| | | |
+-+------------+ +---+------------+ +-----+--------+ +------+---------+
| OPERATIONS | | PLANNING | | LOGISTICS | | FINANCE / |
| SECTION | | SECTION | | SECTION | | ADMIN |
| (Tactics, | | (Tracking, | | (Fuel, Food,| | (Costs, FEMA |
| Valves, O&M)| | IAP Author) | | Equipment) | | Documentation)|
+--------------+ +----------------+ +--------------+ +----------------+
Unified Command
When a water disaster crosses jurisdictional boundaries or impacts multiple statutory agencies (for example, a toxic railcar derailment releasing chemicals into a municipal reservoir), a Unified Command structure is established. Under Unified Command, designated leaders from the water utility, fire department, law enforcement, and environmental protection agencies co-locate in a single command post. Together, they establish a unified set of incident objectives and a single coordinated Incident Action Plan, eliminating jurisdictional conflict while preserving each agency's statutory authorities.
Table 17.4.1: Incident Command System (ICS) Functional Roles and Responsibilities
| ICS Organizational Role | Core Functional Responsibility | Specific Water Treatment Emergency Action |
|---|---|---|
| Incident Commander (IC) | Overall incident authority, establishing objectives, public safety. | Coordinates overall utility disaster response; signs formal emergency declarations. |
| Safety Officer | Monitors hazardous conditions; holds absolute authority to halt unsafe operations. | Halts a chlorine gas entry team if SCBA pressures or Level A chemical suits fail safety checks. |
| Public Info Officer (PIO) | Manages news media; single point of public contact. | Releases official Boil Water Advisories to television, radio, and digital channels. |
| Liaison Officer | Agency coordination with external partners. | Coordinates communications with state primacy regulators and local electric utilities. |
| Operations Section | Directs and executes tactical field operations. | Operates isolation gate valves, throttles pumps, deploys Chlorine Institute Repair Kit B. |
| Planning Section | Collects incident data, tracks resources, drafts IAP. | Maps pressure zone failures, forecasts water storage depletion curves, writes shift IAP. |
| Logistics Section | Procures equipment, materials, fuel, facilities. | Obtains emergency diesel generators, mobile chlorination skids, and replacement filter media. |
| Finance / Admin | Tracks costs, contractor invoices, payroll, claims. | Documents emergency vendor expenses for subsequent FEMA disaster public assistance reimbursement. |
Physical Plant Security and Multi-Barrier Defense
Water utilities apply a defense-in-depth (multi-barrier) engineering framework to secure physical facilities against vandalism, sabotage, theft, and terrorism:
Perimeter and Access Security
- Perimeter Fencing: Facilities must be enclosed by heavy 8-foot industrial chain-link fencing topped with 3-strand barbed wire outriggers angled outward at 45 degrees, or razor wire concertina coils. Vehicle access gates must remain closed and electronically interlocked, utilizing card-swipe or RFID key-fob access.
- Intrusion Detection: Perimeter Intrusion Detection Systems (PIDS), including infrared beams, fence-mounted vibration sensors, and buried seismic cables, alert operators instantly to fence breaches.
- Surveillance and High-Intensity Lighting: Closed-Circuit Television (CCTV) cameras equipped with pan-tilt-zoom (PTZ), thermal night vision, and motion-detection analytics must cover raw water intakes, chemical loading docks, and exterior clearwell areas. High-intensity LED floodlighting (minimum 0.5 to 1.0 foot-candles across general grounds, 2.0 to 5.0 foot-candles at gates and doorways) eliminates unmonitored blind spots.
Finished Water Storage Security
Finished water reservoirs, standpipes, and elevated storage tanks represent the most vulnerable assets in a public water supply, because contaminated water in a storage tank flows directly to consumers without passing through downstream treatment:
- Access Hatches: Must feature raised concrete or steel curbs terminating at least 4 to 6 inches above the tank roof surface to prevent ponding rainwater from leaking into the reservoir. Hatch lids must overlap the raised curb by at least 2 inches, utilize a continuous weather-tight neoprene or elastomeric gasket, and be secured with heavy corrosion-resistant padlocks and tamper-evident security seals. Access hatches should incorporate magnetic intrusion switches tied into SCADA alarms.
- Ventilation Pipe Appurtenances: Tank air vents must point downward (gooseneck or mushroom cowl configuration) and be fitted with a sturdy, corrosion-resistant fine-mesh screen (>= 24-mesh non-corrodible stainless steel or bronze) to prevent the entry of insects, birds, rodents, and airborne particulates.
- Ladders and Elevated Structures: Exterior ladder rungs must terminate at least 8 feet above the finished ground level, or be enclosed within locked steel ladder guards (anti-climb shrouds) to prevent unauthorized access.
Chemical Storage Security
- Chlorine and Hazardous Gas Containment: Liquefied chlorine gas storage rooms and chlorine evaporator galleries must remain locked 24 hours a day with access restricted to certified operators. Rooms must feature crash hardware on exit doors, floor-level exhaust grates routed to automated emergency caustic scrubbers, and gas leak sensors tied to external audio-visual strobes.
- Chemical Off-Loading Protocols: Bulk chemical fill lines (for sodium hypochlorite, liquid alum, caustic soda, hydrofluorosilicic acid) must feature color-coded, labeled, and padlocked connection caps. Plant operators must physically verify shipping manifests, confirm delivery driver identities, and perform verification bench testing (specific gravity, pH, color) before unlocking fill ports.
Table 17.4.2: Water Treatment Plant Physical Security Controls and Standards
| Security Domain | Defense-in-Depth Engineering Specification | Operational Compliance Standard | |---|---|---|\n| Perimeter Barrier | 8-foot industrial chain-link with 3-strand outward barbed wire. | Encloses entire treatment boundary; automated electronic vehicle gates with card access. | | Finished Water Tank Hatches | Raised 4–6 inch curb, overlapping lid, elastomeric gasket, padlocked. | Prevents rainwater and malicious chemical introduction; monitored via SCADA intrusion switches. | | Finished Water Tank Vents | Downward gooseneck cowl equipped with fine-mesh screen. | Must utilize >= 24-mesh stainless steel screen to exclude insects, birds, and airborne debris. | | Tank Access Ladders | Anti-climb ladder guard or rungs terminating >= 8 ft above grade. | Prevents unauthorized trespassers from climbing onto elevated or standpipe storage roofs. | | Hazardous Chemical Rooms | Floor-level exhaust to caustic scrubbers, locked access, gas sensors. | Access restricted; chlorine sensors activate emergency scrubbers at >= 1.0 ppm or >= 3.0 ppm. |
Industrial Control System (SCADA/OT) Cybersecurity
Modern water treatment plants rely entirely on Supervisory Control and Data Acquisition (SCADA) and Operational Technology (OT) systems to monitor water quality sensors, command chemical metering pumps, modulate filter valves, and control high-service finished water distribution pumps. Historically, SCADA networks were physically isolated ("air-gapped") from external telecommunications. However, the modern convergence of enterprise business networks, remote telemetry, automated cellular meter reading, and vendor remote diagnostics has exposed municipal control systems to sophisticated cyber threats:
Primary Cyber Threat Vectors
- Ransomware: Malicious software encrypts plant SCADA servers, human-machine interfaces (HMIs), and historical databases, demanding extortion payments to restore operations.
- Unauthorized Remote Access: Attackers exploit weak passwords, default manufacturer credentials, or unpatched virtual private network (VPN) gateways to access the control interface.
- Process Manipulation: Cyber adversaries alter PLC setpoints, forcing dangerous operational failures: injecting lethal dosages of liquid chlorine or sodium hydroxide (caustic soda), closing distribution valves to cause severe water hammer bursts, or blinding SCADA alarms so operators are unaware of raw water filter breakthrough.
Essential CISA and EPA Cybersecurity Controls
Under guidance from the Cybersecurity and Infrastructure Security Agency (CISA) and the EPA, water treatment utilities must implement rigorous cyber defense controls:
- Strict Network Segmentation: The utility's business Information Technology (IT) network (corporate email, customer billing, internet browsing) must be strictly isolated from the plant's Operational Technology (OT) SCADA network. Traffic between IT and OT must pass through an engineered Demilitarized Zone (DMZ) protected by industrial firewalls. Under no circumstances should a SCADA HMI or PLC be connected directly to the public internet.
- Multi-Factor Authentication (MFA): MFA must be enforced on all remote access connections into the utility network, including vendor remote support links and operator on-call tablets. Single-factor password authentication is strictly prohibited for remote control access.
- Elimination of Default Passwords: All default factory administrative credentials on PLCs, Remote Terminal Units (RTUs), network switches, wireless radios, and SCADA software must be immediately changed to strong, complex passwords managed under Role-Based Access Control (RBAC).
- Immutable Offline (Air-Gapped) Backups: Utilities must maintain regular, verified, and offline backups of SCADA software images, HMI graphics, and PLC logic ladder code. Backups kept online are vulnerable to simultaneous encryption during ransomware attacks.
- Vulnerability Management and Patching: Implement routine testing and deployment of firmware updates and operating system security patches on all SCADA servers, engineering workstations, and networking hardware.
Table 17.4.3: Industrial Control System (SCADA/OT) Cybersecurity Controls
| Cybersecurity Control | Technical Architecture & Implementation | Threat Mitigated |
|---|---|---|
| Network Segmentation | Firewalls and Demilitarized Zones (DMZs) separating corporate IT from plant OT. | Prevents phishing or ransomware on business email from migrating into plant SCADA PLCs. |
| Multi-Factor Authentication | MFA required for all remote VPN access by operators, engineers, and vendors. | Prevents credential theft from granting unauthorized external control over plant machinery. |
| Credential Hardening | Eliminate default factory passwords; enforce complex, unique credentials. | Halts automated brute-force scripts targeting known manufacturer default credentials. |
| Offline Backups | Immutable, air-gapped backups of PLC ladder logic, SCADA configurations, and OS. | Enables rapid bare-metal operational recovery following a catastrophic ransomware lock-out. |
| Intrusion Detection (IDS) | Deploy passive OT-specific network anomaly and packet inspection sensors. | Detects unauthorized PLC write commands, abnormal setpoint adjustments, and network scans. |
Emergency Water Supply Planning, WARN, and Boil Water Advisories
Emergency Drinking Water Provisioning
During catastrophic distribution outages, treatment failures, or severe drought emergencies, water utilities must execute contingency plans to supply emergency potable water to customers. The Federal Emergency Management Agency (FEMA) and the EPA establish that the minimum basic emergency water requirement for human survival is 1 to 2 gallons per person per day (approximately 0.5 to 1.0 gallon/day for drinking and 0.5 to 1.0 gallon/day for minimal sanitation and food preparation). Hospitals, long-term care facilities, and kidney dialysis centers require significantly higher operational allocations.
Water and Wastewater Agency Response Network (WARN)
Utilities maintain membership in their respective state WARN (Water and Wastewater Agency Response Network) program. WARN is a standardized, pre-scripted mutual aid network governed by a signed legal interlocal agreement among public and private water utilities within a state. When an emergency strikes (hurricane, freeze, tornado, catastrophic main failure), a damaged utility can request personnel, emergency generators, high-capacity mobile pumps, chemical supplies, and specialized leak repair crews from other WARN member utilities. Because liability, insurance indemnification, worker compensation, and cost reimbursement structures are legally pre-approved in the master WARN agreement, mutual aid arrives within hours without contractual delays.
Boil Water Advisories (BWAs)
A Boil Water Advisory (BWA) is a formal public health notice issued to water consumers instructing them to vigorously boil all water utilized for drinking, cooking, ice making, teeth brushing, and food preparation:
- Regulatory Trigger: Distribution Pressure Loss: The primary physical trigger for issuing a mandatory Boil Water Advisory is a drop in distribution system pressure below 20 psi (138 kPa). Under standard operating conditions, continuous positive pressure prevents external groundwater, stormwater, and sewer trench leakage from entering water mains. When pressure collapses below 20 psi, a severe hydraulic vacuum forms, causing backsiphonage that draws contaminated soil moisture and non-potable liquids into the distribution system through pipe joints, pinhole leaks, and cracked mains.
- Boiling Instructions: Consumers must bring water to a full, vigorous rolling boil for at least 1 full continuous minute (60 seconds) before consumption. At elevations above 6,500 feet (2,000 meters), water boils at lower temperatures due to reduced atmospheric pressure, requiring water to be boiled for at least 3 continuous minutes.
- Rescinding a Boil Water Advisory: To lift or rescind a BWA, the utility must:
- Restore positive operating pressure throughout the entire distribution network (> 20 psi under peak demand conditions).
- Thoroughly flush distribution mains to evacuate stagnation and air pockets.
- Verify a stable, adequate disinfectant residual (> 0.2 mg/L free chlorine or chloramines) throughout the affected pressure zone.
- Collect representative microbiological water samples throughout the affected area that test negative for total coliform bacteria and E. coli for two consecutive days (two sets of samples collected 24 hours apart), or satisfy state primacy criteria for release.
Under Section 2013 of America's Water Infrastructure Act (AWIA) of 2018, which public water systems are required to prepare a Risk and Resilience Assessment (RRA) and Emergency Response Plan (ERP), and on what recurring schedule must they be updated and certified?
During a major chemical emergency involving a chlorine gas release at a water treatment plant, an Incident Command System (ICS) structure is established. Which Command Staff member has the explicit, autonomous authority to immediately halt operations or change tactical procedures to protect personnel?
An industrial cyberattack attempts to compromise a water treatment facility's SCADA programmable logic controllers (PLCs) to manipulate chemical feed setpoints. Which combination of cybersecurity architecture and emergency operational response represents industry best practice?