10.3 Factory Acceptance Testing (FAT): Planning, Test Protocols, Execution & Punch Lists

Factory Acceptance Testing (FAT): Planning, Test Protocols, Execution & Punch Lists

In Domain 4 (Development) of the ISA Certified Automation Professional body of knowledge, the Factory Acceptance Test (FAT) represents the formal contractual and technical gate separating system development from physical site deployment. The FAT is conducted at the system integrator’s or vendor’s staging facility in a controlled environment prior to packaging, crating, and shipping equipment to the customer site.

From a project management and lifecycle economics perspective, detecting and correcting errors during the FAT is orders of magnitude less expensive than rectifying them during field commissioning. The rule of ten, a widely used quality-cost heuristic, holds that an architectural flaw or coding error costing $100 to correct during staging will cost $1,000 to remediate during site installation, and upwards of $10,000 to $100,000 to fix if discovered after process fluids have been introduced into the plant.

---

FAT Prerequisites: Entry Criteria

A disciplined automation project enforces strict entry criteria before convening client engineers, operations representatives, and third-party inspectors for a witnessed FAT. Attempting to begin an FAT with incomplete code or unverified hardware wastes project resources, damages professional credibility, and results in abortive testing.

FAT LIFECYCLE GATES & PREREQUISITE PROGRESSION:

+-------------------------------------------------------------------------+
| PREREQUISITE GATE: DOCUMENTATION & CODE MATURITY                        |
| - Approved Functional Design Specification (FDS) & Detailed Design (DDS)|
| - Approved Control Schematics, Panel Layouts & Loop Sheets              |
| - Application Software Code 100% Compiled & Static Lint Check Passed    |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| SYSTEM INTEGRATOR INTERNAL PRE-FAT (DRY RUN)                            |
| - 100% internal checkout of all I/O, sequences, and redundancy          |
| - Internal defect logging and resolution (zero known major flaws)       |
| - NIST-traceable calibration certificates verified for all test gear    |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| CUSTOMER-WITNESSED FAT EXECUTION (STAGING FACILITY)                     |
| - Step-by-step execution of approved FAT Protocol                      |
| - Hardware inspection, 100% I/O simulation, failovers, C&E interlocks   |
| - Real-time punch list categorization (Category A, B, C)                |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| FAT COMPLETION GATE & SHIPPING RELEASE CRITERIA                         |
| - Category A: 100% Resolved and Re-tested                               |
| - Category B: Agreed remediation plan & schedule for site SAT           |
| - Formal Certificate of Completion signed by Customer & Integrator      |
+-------------------------------------------------------------------------+

Mandatory FAT Prerequisites Checklist

1. Approved Design Baselines: The Functional Design Specification (FDS), Software Design Specification (SDS), Hardware Design Specification (HDS), electrical schematics, panel fabrication drawings, and Process & Instrumentation Diagrams (P&IDs) must have "Approved for Construction" (AFC) or "Approved" status.
2. Approved FAT Test Protocol: The step-by-step test procedure document must be formally reviewed and approved by the customer/owner engineering team at least 2 to 4 weeks prior to testing. Ad-hoc, unscripted testing is strictly prohibited.
3. Completed Application Code: Application software (PLC/PAC logic, safety system logic solver code, SCADA/HMI graphic screens, database configurations, and historical alarm databases) must be $100\%$ complete.
4. Internal Pre-FAT "Dry Run": The system integrator must execute the entire FAT protocol internally without the client present. All discovered bugs, wiring errors, and configuration flaws must be corrected and re-tested before welcoming the client to the staging floor.
5. Staging Environment Readiness: Control panels must be fully mounted, illuminated, grounded, labeled, and powered via regulated utility sources with secondary backup. Signal simulators, digital toggle boxes, and $4\text{--}20\text{ mA}$ calibrators must be in place.
6. Calibrated Test Equipment: Multimeters, loop calibrators, decibel meters, oscilloscope instruments, and network protocol analyzers must possess valid, unexpired NIST-traceable (or national equivalent) calibration certificates.

---

FAT Protocol Structure and Execution Methodology

An FAT protocol is an auditable quality and testing document. It must be organized into modular, unambiguous test cases designed so that any qualified automation engineer can execute the steps and obtain identical results.

Standard Protocol Test Case Format

Every test script within the FAT document must contain the following formal data blocks:

+-------------------------------------------------------------------------+
| TEST SCRIPT: TC-INT-042: High-High Reactor Pressure Emergency Depress.  |
+-------------------------------------------------------------------------+
| Objective: Verify SIL 2 safety interlock I-101 de-energizes dump valve  |
|            XV-101 and triggers Level 1 emergency alarm upon PT-101 > 85 |
+-------------------------------------------------------------------------+
| Prerequisite Setup:                                                     |
| 1. System in RUN mode with no active bypasses.                          |
| 2. Fluke 789 calibrator connected to Field Terminal TB2-14/15 (PT-101). |
| 3. Reactor base pressure simulated at normal 50.0 psig (12.0 mA).       |
+-------------------------------------------------------------------------+
| Step | Action Taken               | Expected Result       | Pass | Fail |
+------+----------------------------+-----------------------+------+------+
| 1    | Increase PT-101 to 84.5 psi| Graphic shows 84.5 psi| [X]  | [ ]  |
|      | (17.52 mA).                | XV-101 remains closed.|      |      |
| 2    | Increase PT-101 to 85.5 psi| Red alarm banner fires| [X]  | [ ]  |
|      | (17.68 mA).                | XV-101 drops to 0 VDC |      |      |
|      |                            | Solenoid de-energizes |      |      |
| 3    | Verify HMI First-Out log.  | I-101 logged at ms res| [X]  | [ ]  |
+-------------------------------------------------------------------------+
| Actual Results / Comments: XV-101 dropped out at exactly 85.1 psig.     |
| Verified on staging digital output terminal TB4-08.                     |
+-------------------------------------------------------------------------+
| Customer Witness:   J. Doe (P.Eng)     Date: 2026-10-12   Result: PASS  |
| Integrator Lead:    K. Lee (CAP)       Date: 2026-10-12                 |
+-------------------------------------------------------------------------+

---

Scope of Testing during FAT

A rigorous FAT encompasses both hardware build verification and dynamic software functional testing.

1. Panel Physical and Electrical Build Inspection

• Mechanical Dimensions & Finish: Verifying enclosure dimensions, paint thickness, door swing clearances, plinth mounting holes, eye bolt ratings, and ingress protection (e.g., NEMA 4/12 or IP54/65).
• Thermal Management & Environmental Controls: Inspection of panel roof exhaust fans, louvers, air conditioner units, internal thermostats, and enclosure anti-condensation heaters.
• Wire Segregation & Routing: Ensuring strict physical separation between high-voltage AC power distribution ($120/240/480\text{ VAC}$), low-voltage DC power/instrumentation ($24\text{ VDC}$), sensitive analog signals, and fiber-optic/Ethernet communications in separate, dedicated plastic wire ducts. Contract specifications commonly require AC and DC wiring to run in separate ducts.
• Ferrule Numbering & Color Coding: Verifying that wire insulation colors conform to contract standards (e.g., NFPA 79: Black for ungrounded AC line, Red for AC control, Blue for ungrounded DC control, White with Blue stripe for DC common, Green with Yellow stripe for protective earth). Every wire must bear heat-shrink or sleeve-type printed ferrules matching panel wiring schematics.
• Grounding Integrity: Measuring electrical bonding resistance with a low-resistance ohmmeter. Grounding studs on cabinet doors, side panels, equipment subplates, and the master instrument protective ground bar must exhibit an impedance of less than $0.1\ \Omega$.

2. 100% Hardwired I/O Loop Simulation

Every single I/O channel on every module must be verified individually from the physical field terminal blocks through to the controller memory and HMI graphical display. Statistical or "spot-check" sample testing (such as checking only $10\%$ of channels) is strictly unacceptable for an industrial FAT.

3. Redundancy Failover Testing

Modern automation architectures incorporate redundancy to eliminate single points of failure. The FAT must rigorously validate that failover transitions occur seamlessly without disturbing process control:

• Power Supply Redundancy: Disconnect Primary AC Feed A. Verify that redundant power supplies and diode auctioneering modules maintain seamless $24\text{ VDC}$ bus voltage without a millivolt dip. Restore Feed A and disconnect Feed B. Verify diagnostic alarms trip on the HMI for each power supply failure.
• Controller Redundancy Switchover: With the controller executing active sequences and regulatory loops under full simulated I/O communication traffic, the primary CPU is powered down, halted, or its redundancy fiber synchronization cable is physically severed. The test verifies:
  1. The secondary standby CPU assumes control bumplessly within the specified window (typically $< 10\text{ to } 20\text{ ms}$).
  2. Field outputs do not drop out, chatter, or momentarily glitch to zero.
  3. Active batch sequences, timers, and PID loops continue execution without resetting.
  4. HMI displays indicate the switchover event and generate a critical diagnostic alarm.
• Network Infrastructure Redundancy: Sever one segment of a Device Level Ring (DLR) or Media Redundancy Protocol (MRP) industrial Ethernet ring. Measure the network recovery time using a network analyzer (verifying that the ring recovers within its specified time—typically a few milliseconds for DLR, or the configured recovery class for MRP) and confirm zero lost I/O packets or rack connection drops.

4. Sequence Logic, Interlocks & Cause-and-Effect Matrix Verification

• Cause-and-Effect (C&E) Matrix Verification: 100% testing of all safety instrumented functions (SIF) and process interlocks. Every defined cause (initiating event) must be systematically triggered to verify that every specified effect (final control element trip, valve closure, pump shutdown, audible horn) occurs strictly per the matrix.
• Batch Sequence Step-Through: Exercising automated phases per ISA-88 across all standard states: Run, Hold, Restart, Stop, and Abort.
• Alarm System Rationalization Verification: Verifying alarm setpoints, priorities, audible tones, color schemes (per ISA-18.2 / ISA-101), shelving functions, and first-out trip detection.

---

Punch List (Deficiency List) Categorization

Inevitably, discrepancies, software defects, hardware wiring errors, and documentation gaps will be discovered during the execution of an FAT. These items are formally recorded in the FAT Punch List (Deficiency Log).

To prevent subjective debates regarding whether a system is fit for delivery, punch list items are categorized into tiers; a common three-tier scheme is:

PUNCH LIST DEFICIENCY CLASSIFICATION:

+-------------------------------------------------------------------------+
| CATEGORY A (CRITICAL / BLOCKING DEFICIENCY)                             |
| - Fatal safety flaw, missing mandatory FDS requirement, CPU failover    |
|   failure, damaged structural enclosure, non-functional safety interlock|
|                                                                         |
| ACTION: HARD STOP. Equipment CANNOT be shipped.                         |
|         Must be remediated and re-tested at staging facility.           |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| CATEGORY B (MAJOR / NON-BLOCKING DEFICIENCY WITH WORKAROUND)            |
| - Secondary graphic animation defect, non-critical diagnostic alarm,    |
|   minor logic bug with approved temporary workaround, missing spare tag |
|                                                                         |
| ACTION: SHIPMENT PERMITTED with written customer variance agreement.    |
|         Remediation scheduled for site pre-commissioning or SAT.        |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| CATEGORY C (MINOR / COSMETIC / DOCUMENTATION DEFICIENCY)                |
| - Drawing redline markup, typographical error in O&M manual, misaligned |
|   panel label, minor paint scratch on non-corrosive interior subplate   |
|                                                                         |
| ACTION: SHIPMENT PERMITTED.                                             |
|         Resolved prior to final plant handover and as-built turnover.   |
+-------------------------------------------------------------------------+

---

FAT Certificate of Completion and Shipping Release Criteria

The Factory Acceptance Test officially concludes only when the formal FAT Certificate of Completion (Shipping Release) is signed by authorized representatives of the customer, the engineering procurement contractor (EPC), and the system integrator.

Mandatory Shipping Release Criteria:

1. Zero Open Category A Items: Every single Category A punch list item must be resolved, regression tested, and re-signed as PASSED by the client witness. Under no circumstances may a system be shipped with an open Category A flaw.
2. Agreed Category B Action Plan: Every Category B item must have a documented root-cause remediation plan, an assigned responsible engineer, and an agreed target completion milestone (e.g., prior to Site Acceptance Test initiation).
3. As-Tested Software Baseline Secured: The exact software binaries, logic programs, and HMI runtime files executed during the FAT must be compiled, checksummed, backed up to secure revision-controlled media, and archived.
4. Panel De-Staging and Crating Inspection: Panels must be powered down, disconnected, desiccated with silica gel packets, sealed in vapor-barrier poly-wrap, and secured in vibration-damped wooden shipping crates.

---

Realistic Industrial Scenario: Pharmaceutical WFI Skid Control System FAT

An automation engineering contractor executes a customer-witnessed FAT for a Water for Injection (WFI) generation skid at their staging facility. The skid is governed by an Allen-Bradley ControlLogix PAC paired with an Endress+Hauser instrumentation package.

1. Day 1: Hardware Inspection & 100% I/O Checkout: The client electrical engineer discovers that $120\text{ VAC}$ solenoid power wiring is routed through the same Panduit wire duct as shielded $4\text{--}20\text{ mA}$ temperature transmitter cables.
   • Outcome: Logged as Punch List Item 01 (Category A). Violates the contract wiring-segregation specification and introduces induced noise. Integrator rewires and pulls the $120\text{ VAC}$ circuits into an isolated duct overnight. Re-inspected and closed on Day 2.
2. Day 2: Redundancy Failover: The technician unplugs the fiber-optic redundancy module on Primary CPU A. The secondary CPU takes control within $8\text{ ms}$. All sanitary diaphragm control valves maintain steady position; the client witnesses the event and initials the test script as PASS.
3. Day 3: Sanitization Sequence & Cause-and-Effect Matrix: During the automated $121^{\circ}\text{C}$ steam sanitization sequence, the high-temperature hold timer times out, but the sequence fails to advance to the cooling blowdown step because an unmapped auxiliary contact bit was configured as a transition permissive.
   • Outcome: Logged as Punch List Item 02 (Category A). Logic corrected, peer-reviewed, downloaded, and the complete sanitization cycle re-tested successfully.
4. Day 4: HMI Screen Review: The client notes that the conductivity readout is displayed to four decimal places (0.0542 uS/cm), which causes visual flicker on the HMI screen; two decimal places was requested.
   • Outcome: Logged as Punch List Item 03 (Category B). Workaround accepted; client agrees this can be updated during SAT graphic loading.
5. Closure: All Category A items are cleared. The client project manager signs the FAT Certificate of Completion, releasing the panels for shipping to the pharmaceutical facility.

---

Common Exam Traps

1. Authorizing Shipment with Open Category A Punch List Items: A classic exam question scenario involves project schedule pressure tempting an engineer to ship equipment with an open Category A item, intending to "fix it on site." This is a catastrophic breach of engineering ethics and quality governance. Category A defects strictly block shipment.
2. Confusing FAT with SAT (Site Acceptance Testing):
   • FAT: Performed at the vendor/integrator staging facility using simulated I/O and temporary power.
   • SAT: Performed at the actual customer industrial site after physical mounting, final field wiring termination to real instruments, and connection to permanent plant power.
3. Conducting a Witnessed FAT without an Internal Pre-FAT Dry Run: Exam scenarios frequently describe an FAT where the client arrives and encounters dozens of basic wiring and coding errors on Day 1. The root cause is invariably the omission of the integrator's internal pre-FAT dry run.
4. Accepting Statistical Sample Testing for Hardwired I/O: While high-volume consumer goods rely on statistical batch sampling, mission-critical industrial automation systems require 100% point-to-point hardwired I/O testing during the FAT.