16.2 Quality Assurance & Continuous Quality Improvement (APCO/NENA ANS 1.107)

16.2 Quality Assurance & Continuous Quality Improvement (APCO/NENA ANS 1.107)

Quick Answer: A defensible public safety quality management program pairs retrospective Quality Assurance (QA), auditing call handling against established standards, with prospective Continuous Quality Improvement (CQI), using the findings to improve workflows, training, and coaching. The governing standard is APCO/NENA ANS 1.107.2-2025, which replaced ANS 1.107.1-2015; do not confuse it with NENA-STA-020, NENA's call processing and answering standard. ANS 1.107.2 scales random review to center size (about 3% of calls for service in small centers, 2% in medium centers, and 1% in large centers), requires review of all high-acuity calls (such as cardiac arrests, officer emergencies, and working structure fires) with reviews starting within a few days, and calls for standardized evaluation forms and timely feedback. Agencies keep scoring consistent through evaluator calibration and respond to findings within a non-punitive Just Culture framework that distinguishes human error, at-risk behavior, and reckless behavior.

---

1. QA vs. CQI Architecture & APCO/NENA ANS 1.107 Governance

While often used interchangeably, Quality Assurance (QA) and Continuous Quality Improvement (CQI) represent distinct, complementary operational functions in public safety communications:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     QUALITY ASSURANCE VS. QUALITY IMPROVEMENT               │
├──────────────────────────────┬──────────────────────────────────────────────┤
│ QUALITY ASSURANCE (QA)       │ CONTINUOUS QUALITY IMPROVEMENT (CQI)         │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ • Retrospective audit focus  │ • Prospective optimization focus             │
│ • Measures compliance to SOP │ • Redesigns workflows and eliminates barriers│
│ • Individual-level scoring   │ • Systemic, agency-wide trend analysis       │
│ • Question: "Did the worker  │ • Question: "Why did the system permit this  │
│   follow the standard?"      │   failure, and how can we prevent recurrence?"│
│ • Primary tool: Scorecards   │ • Primary tool: Coaching, training, redesign │
└──────────────────────────────┴──────────────────────────────────────────────┘

The National Consensus Standard: APCO/NENA ANS 1.107

The recognized benchmark for public safety quality programs is the APCO/NENA American National Standard for establishing a Quality Assurance and Quality Improvement (QA/QI) program. The first edition, ANS 1.107.1-2015, has been replaced by ANS 1.107.2-2025. Its core elements include:

1. Designated Oversight: A written QA/QI program with trained, designated personnel responsible for case review.
2. Standardized Evaluation Instruments: Published evaluation forms with clear definitions across call-taking and dispatch disciplines (law enforcement, fire, EMS).
3. Timely Review and Feedback: Reviews and feedback delivered soon after the call; for high-acuity calls, ANS 1.107.2 expects review to begin within a few days (3 to 5), because delayed feedback loses its teaching value and leaves unsafe habits uncorrected.
4. Closed-Loop Improvement: Findings feed coaching, training updates, and changes to policies or systems, with the follow-up documented.

---

2. Call Sampling Methodologies: Random vs. Mandatory Targeted Auditing

A legally defensible and operationally sound quality program cannot rely on ad-hoc or supervisor-selected calls. It must balance broad statistical sampling with dedicated scrutiny of catastrophic liability incidents:

                      PUBLIC SAFETY QA SAMPLING ARCHITECTURE
                                         │
         ┌───────────────────────────────┴───────────────────────────────┐
         ▼                                                               ▼
SIZE-BASED RANDOM SAMPLING (ANS 1.107.2)                     HIGH-ACUITY & TARGETED REVIEWS
• Small centers (<=100k calls/yr): about 3%                  • All high-acuity calls (e.g., cardiac arrest)
• Medium centers: about 2%                                   • Officer distress / Mayday calls
• Large centers (500k+ calls/yr): about 1%                   • Working structure fires
• Stratified across shifts & call types                      • Vehicle pursuits (per agency policy)
• Eliminates reviewer cherry-picking                         • Formal citizen / agency complaints

Size-Based Random Sampling

To measure baseline compliance, the QA program reviews a representative cross-section of day-to-day operations:

• Sampling Levels (ANS 1.107.2-2025): About 3% of calls for service for centers handling 100,000 or fewer per year, 2% for centers handling between 100,000 and 500,000, and 1% for centers handling 500,000 or more. The 2015 edition set a general minimum of 2%.
• Stratification Across Operations: Random selection should span shifts (day, swing, midnight), days of the week, and call types (9-1-1 voice, 10-digit lines, text-to-911). Evaluators should never cherry-pick calls based on personal preference or a telecommunicator's reputation.

High-Acuity and Targeted Reviews

Random sampling will miss rare, high-consequence incidents, so the standard requires review of all high-acuity calls, and agencies add targeted reviews for other critical categories:

---

3. Structured QA Rubrics & Core Scoring Domains

Objective evaluation requires structured rubrics that assess observable operational behaviors rather than subjective impressions. Many agency QA forms group call handling into five core scoring domains:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     CORE PUBLIC SAFETY QA SCORING DOMAINS                   │
├──────────────────────────┬──────────────────────────┬───────────────────────┤
│  LOCATION VERIFICATION   │  CALL CONTROL & TRIAGE   │  PROTOCOL ADHERENCE   │
│       (Domain 1)         │       (Domain 2)         │      (Domain 3)       │
├──────────────────────────┼──────────────────────────┼───────────────────────┤
│ • Exact address & cross  │ • Answer within 15 sec   │ • Verbatim scripted   │
│ • Apt / suite / building │ • Professional greeting  │   pre-arrival delivery│
│ • Coordinate validation  │ • Repetitive persistence │ • Correct chief complaint│
│ • ALI re-bid execution   │ • Active de-escalation   │ • Shunt navigation   │
└──────────────────────────┴─────────────┬────────────┴───────────────────────┘
                                         ▼
                 ┌───────────────────────────────────────────┐
                 │ CAD TIMELINESS & ACCURACY (Domain 4)      │
                 │ • Queue dispatch latency ≤30 seconds      │
                 │ • Proper response beat and priority code  │
                 │ • Hazardous officer safety remarks logged │
                 ├───────────────────────────────────────────┤
                 │ CUSTOMER SERVICE & TONE (Domain 5)        │
                 │ • Professional, empathetic demeanor       │
                 │ • Freedom from sarcasm / argumentative tone│
                 │ • Reassurance and scene safety guidance   │
                 └───────────────────────────────────────────┘

The Five Core Scoring Domains

1. Location Verification (The Non-Negotiable Core): Location is the paramount operational element of any emergency call. Telecommunicators must verify the exact physical street address, building/suite/apartment number, business name, nearest intersecting cross streets, and caller callback number. On wireless calls, they must re-bid ALI and validate raw coordinates against GIS mapping. Zero-Tolerance Standard: Entering an unverified or incorrect address is classified as a critical operational failure requiring immediate remediation.
2. Call Control & Caller Management: Supporting the 15-second answering benchmark (NENA-STA-020 and NFPA 1225), delivering authorized agency greetings, maintaining command over hysterical callers using repetitive persistence (repeating instructions using identical phrasing and a calm, firm tone), and filtering extraneous noise.
3. Protocol Adherence (Priority Dispatch Logic): Navigating structured triage algorithms (EMD, EFD, EPD, or law protocols) without omitting mandatory questions or freelancing unapproved instructions. In medical crises, delivering scripted pre-arrival instructions (CPR, choking relief, severe hemorrhage control, emergency childbirth) verbatim without hesitation.
4. CAD Timeliness & Accuracy: Entering incidents into the dispatch queue rapidly (target ≤30 to 60 seconds for high-priority emergencies), assigning accurate incident type codes, selecting correct response beats, and continuously updating CAD with tactical officer safety warnings.
5. Customer Service & Tone: Projecting professional empathy, validating caller distress, maintaining composure in the face of verbal abuse, and balancing protocol discipline with human reassurance.

---

4. Compliance Scoring Thresholds & High-Reliability Accreditation

Agencies establish compliance thresholds that distinguish acceptable operational performance from sub-standard execution requiring intervention. Quality scores are evaluated at both the individual telecommunicator level and the agency-wide system level.

Protocol Accreditation Benchmarks

Agencies using structured protocol systems may pursue accreditation through the protocol vendor, such as the International Academies of Emergency Dispatch (IAED) Accredited Center of Excellence (ACE) program. These programs set their own compliance thresholds for elements such as chief complaint selection, key questions, determinant coding, and pre-arrival instructions, and they require ongoing case review data. Check the current accreditation criteria rather than memorizing fixed percentages, which vary by program and protocol.

Categorizing Protocol Deviations

Modern QA scoring instruments differentiate between minor procedural variances and critical life-safety non-compliance:

• Non-Critical Variance: Minor phrasing alterations that do not distort meaning, slight sequencing variations that do not delay response, or brief delays in secondary data entry.
• Critical Non-Compliance: Complete omission of mandatory life-safety questions (e.g., failing to ask if weapons are present), failing to deliver immediate CPR instructions to an arrest victim, or entering an unverified address into CAD. A critical non-compliance error typically results in an automatic failing score for that call evaluation and triggers mandatory closed-loop coaching.

---

5. Evaluator Calibration & Inter-Rater Reliability (IRR)

The most pervasive threat to a QA program's credibility is inter-rater variability—the condition where different evaluators grade the identical call with widely disparate scores. If Evaluator A awards a call a 96% while Evaluator B awards the identical call a 78%, the program loses legitimacy, generating employee cynicism, union grievances, and administrative appeals.

                    EVALUATOR CALIBRATION PROCESS CYCLE
┌─────────────────────────────────────────────────────────────────────────────┐
│ 1. CALL SELECTION: Quality Manager selects 2 to 3 blind operational calls   │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 2. INDEPENDENT SCORING: All QA reviewers, CTOs, and supervisors grade the   │
│    calls independently using the standard rubric without consultation       │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 3. CONVENE CALIBRATION SESSION: Reviewers compare line-item scores          │
│    and debate discrepancies in rubric interpretation                        │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 4. CONSENSUS ALIGNMENT: Establish definitive benchmark score and update     │
│    written QA policy guidelines to clarify ambiguous criteria               │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 5. STATISTICAL VALIDATION: Track agreement against the agency target        │
└─────────────────────────────────────────────────────────────────────────────┘

Conducting Calibration Sessions

To achieve statistical consistency, agencies hold regular evaluator calibration sessions (many do so monthly):

• Blind Independent Grading: The QA coordinator distributes two to three recorded calls representing varying complexity. All QA evaluators, CTOs, and shift supervisors independently review the audio and CAD logs, completing standard scorecards without consulting each other.
• Discrepancy Analysis: The team convenes to review line-item discrepancies. If one evaluator deducted points for "loss of call control" while another awarded full points, the team examines the objective behavior against published rubric definitions.
• The Statistical Benchmark: Agencies set an inter-rater reliability (IRR) target in their QA policy and track it with percentage agreement or Cohen's kappa. There is no single national IRR percentage, so the local target and the trend over time matter most.
• Rubric Clarification: When recurring scoring splits occur, the agency publishes an official Quality Clarification Bulletin to formally refine scoring definitions.

---

6. Just Culture Principles & Non-Punitive Coaching

Historically, public safety operated under a punitive "blame-and-punish" mindset. When an error occurred, management sought someone to discipline. Modern safety science proves that punitive cultures do not eliminate errors; they merely incentivize staff to hide mistakes, falsify logs, and avoid reporting close-calls.

To foster a true high-reliability learning organization, agencies adopt the Just Culture framework (pioneered by David Marx):

┌─────────────────────────────────────────────────────────────────────────────┐
│                     THE JUST CULTURE BEHAVIORAL MATRIX                      │
├──────────────────────┬──────────────────────────┬───────────────────────────┤
│ BEHAVIOR TYPE        │ DEFINITION & EXAMPLE     │ PROPER MANAGEMENT RESPONSE│
├──────────────────────┼──────────────────────────┼───────────────────────────┤
│ 1. HUMAN ERROR       │ • Inadvertent slip,      │ • CONSOLE & SUPPORT       │
│    (Inadvertent)     │   lapse, or mistake      │ • Systemic redesign       │
│                      │ • Ex: Transposing numbers│ • Improve CAD auto-fill   │
│                      │   during severe surge    │ • Address console fatigue │
├──────────────────────┼──────────────────────────┼───────────────────────────┤
│ 2. AT-RISK BEHAVIOR  │ • Behavioral drift       │ • COACH & RE-EDUCATE      │
│    (Unconscious Drift│ • Choice where risk is   │ • Remove incentives for   │
│     or Justified)    │   mistakenly thought     │   the shortcut            │
│                      │   justified or minimal   │ • Re-align operational    │
│                      │ • Ex: Skipping 2nd       │   pressures               │
│                      │   address verification to│                           │
│                      │   answer holding calls   │                           │
├──────────────────────┼──────────────────────────┼───────────────────────────┤
│ 3. RECKLESS BEHAVIOR │ • Conscious, deliberate  │ • FORMAL DISCIPLINE       │
│    (Conscious Risk)  │   disregard of a         │ • Administrative sanctions│
│                      │   substantial, unjustified│ • Suspension or immediate │
│                      │   risk                   │   termination             │
│                      │ • Ex: Hanging up on 9-1-1│                           │
│                      │   callers; sleeping on   │                           │
│                      │   duty; faking CAD logs  │                           │
└──────────────────────┴──────────────────────────┴───────────────────────────┘

The Three Behaviors in Emergency Communications

1. Human Error (Inadvertent Slips or Lapses): An unintentional mistake occurring during normal operations. Example: A call taker inadvertently transposes two digits of a phone number (typing 555-0182 instead of 555-0128) while handling a screaming caller during a severe storm. Management Response: Console the employee, review environmental factors (fatigue, lighting, noise), and evaluate system design (e.g., automated CAD phone lookups).
2. At-Risk Behavior (Behavioral Drift & Shortcuts): A choice where the employee drifts into an unsafe shortcut, either failing to recognize the risk or mistakenly believing the risk is justified. Example: A call taker routinely skips secondary address verification during call surges to reduce handle times and answer waiting 9-1-1 lines faster. Management Response: Coach the employee, re-educate on the catastrophic risk of address errors, and eliminate organizational incentives that reward the shortcut (e.g., removing arbitrary handle time penalties).
3. Reckless Behavior (Deliberate Disregard of Substantial Risk): A conscious, willful choice to disregard a substantial and unjustifiable risk. Example: A telecommunicator intentionally disconnects a difficult caller, sleeps on duty while assigned to an active radio channel, or falsifies CAD records to conceal delayed unit dispatches. Management Response: Immediate punitive administrative sanctions, disciplinary suspension, or termination of employment.

Closing the CQI Feedback Loop

A mature quality program closes the operational loop. Aggregate monthly QA scores are synthesized into executive dashboards. If quality metrics reveal that 30% of telecommunicators score below standard on active assailant suspect sequencing, management recognizes that this is not an individual personnel failure—it is a systemic training deficit. The data is routed directly to the Training Division to update academy curricula, deploy remedial simulator drills, and recalibrate CTO coaching on the dispatch floor.