15.2 Program Evaluation, CQI Methodologies & Outcome Metrics

15.2 Program Evaluation, CQI Methodologies & Outcome Metrics

Quick Answer: Program evaluation in asthma care relies on Continuous Quality Improvement (CQI) frameworks to systematically measure and optimize clinical, functional, and economic outcomes. Grounded in the Donabedian triad—Structure (system inputs, AE-C staffing, EHR capabilities), Process (evidence-based care delivery, written AAP distribution, return inhaler technique checks), and Outcome (quantifiable patient health results such as ACT scores, ED visits, and hospitalizations)—programs evaluate efficacy using validated survey instruments (ACT, ACQ, PACQLQ) and HEDIS measures (Asthma Medication Ratio [AMR] ≥0.50). Rapid, iterative organizational enhancements are operationalized through Plan-Do-Study-Act (PDSA) cycles to eliminate clinical variances and justify healthcare investments.

Developing an asthma education service is only the initial step in chronic disease management; demonstrating that the service consistently improves patient outcomes, adheres to clinical practice guidelines, and delivers cost-effective care is critical for long-term viability. Healthcare systems, regulatory bodies, public health agencies, and commercial payers demand rigorous program evaluation. For Certified Asthma Educators (AE-C) in leadership or clinical coordination roles, mastering quality improvement (QI) science is essential to elevate patient care and justify program resources.

---

Continuous Quality Improvement and the Donabedian Model

Continuous Quality Improvement (CQI) is an ongoing, data-driven philosophy of management that analyzes healthcare processes to identify variations, eliminate inefficiencies, and systematically elevate patient safety and clinical outcomes. CQI rejects punitive blame models, recognizing that over 85% of medical errors and poor outcomes stem from faulty clinical system designs rather than individual clinician negligence.

The gold standard conceptual model for evaluating healthcare quality was developed by physician Avedis Donabedian (1966). The Donabedian Model posits that quality assessment must evaluate three interdependent domains: Structure, Process, and Outcome.

┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐
│    STRUCTURE    │  ──>  │     PROCESS     │  ──>  │     OUTCOME     │
│ System Inputs,  │       │ Care Delivery,  │       │ Health Results, │
│ Staffing & Tech │       │ AAPs, & Checks  │       │ ED/Inpt & ACT   │
└─────────────────┘       └─────────────────┘       └─────────────────┘

1. Structural Measures (System Inputs and Resources)

Structural measures examine the context and environment in which care is delivered, including physical facilities, equipment, human resources, and organizational attributes:

• Ratio of AE-C certified clinicians (RNs, RTs, Pharmacists) to active asthma patient panel size.
• Availability of point-of-care diagnostic tools (calibrated spirometers meeting ATS/ERS standards, disposable bacterial filters, FeNO analyzers, peak flow meters).
• Availability of demonstration teaching devices (placebo MDIs, dry powder inhalers, soft mist inhalers, valved holding chambers with pediatric masks).
• Electronic Health Record (EHR) infrastructure, including structured, discrete data fields for Asthma Control Test (ACT) scores, personal best peak expiratory flow, and trigger tracking.
• Availability of culturally tailored educational collateral translated into primary patient languages at a 4th- to 6th-grade reading level.

2. Process Measures (Clinical Actions and Care Delivery)

Process measures evaluate what clinicians actually do to provide care and whether clinical actions align with evidence-based practice guidelines (NAEPP EPR-3/2020 and GINA):

• Program Reach and Penetration: The percentage of diagnosed asthmatic individuals in the healthcare network referred to and enrolled in structured asthma education.
• Attendance and Completion Rates: The proportion of enrolled participants who complete the prescribed educational sequence (e.g., attending ≥3 sessions).
• Written Asthma Action Plan Distribution: The percentage of active asthma patients who receive an individualized, updated, written Asthma Action Plan within the past 12 months.
• Direct Return Inhaler Demonstration: The proportion of clinical encounters where the educator directly observes and scores the patient's physical inhaler and spacer technique using an objective checklist.
• Tobacco Screening & Counseling: Documented screening for combustible and electronic tobacco exposure, along with delivery of the 5 A's cessation intervention.
• Pharmacy Claims Process Measures: Tracking the Healthcare Effectiveness Data and Information Set (HEDIS) Asthma Medication Ratio (AMR) and Proportion of Days Covered (PDC ≥75%).

3. Outcome Measures (Patient Health Status and System Impact)

Outcome measures evaluate the concrete clinical, functional, and economic results of the care delivered:

• Clinical Metrics: Forced expiratory volume in 1 second (FEV1 % predicted), bronchodilator reversibility, and frequency of oral corticosteroid (OCS) burst prescriptions.
• Healthcare Utilization: Annualized asthma-related Emergency Department (ED) visits per 100 patient-years, inpatient hospital admissions, pediatric intensive care unit (PICU) admissions, and 30-day all-cause/asthma-specific readmission rates.
• Patient-Reported Outcome Measures (PROMs):
  • Asthma Control Test (ACT): A 5-item validated tool evaluating symptom frequency, rescue use, and self-assessed control over the past 4 weeks (scores range 5–25; score ≤19 denotes uncontrolled asthma; Minimal Clinically Important Difference [MCID] is ≥3 points).
  • Asthma Control Questionnaire (ACQ): Scores range 0 (fully controlled) to 6 (severely uncontrolled); score ≥1.5 indicates uncontrolled disease; MCID is 0.5 points.
  • Quality of Life Instruments: Pediatric Asthma Caregiver's Quality of Life Questionnaire (PACQLQ) and Mini Asthma Quality of Life Questionnaire (Mini-AQLQ); MCID is 0.5 points on a 7-point scale.
• Societal & Functional Metrics: Number of missed school days per child and missed work days per adult patient or family caregiver.
• Economic Metrics: Total cost of care per member per month (PMPM), return on investment (ROI), and hospital uncompensated care cost avoidance.

---

Comprehensive Asthma Program Evaluation Metric Matrix

---

Step-by-Step Plan-Do-Study-Act (PDSA) Cycle in Asthma Practice

The Plan-Do-Study-Act (PDSA) cycle (the Deming/Shewhart cycle) is the primary engine of rapid-cycle improvement in healthcare. Rather than spending months planning massive, unverified institutional reorganizations, QI teams design small, rapid tests of change, evaluate quantitative data, and refine interventions iteratively.

Operational Template: Improving Post-Hospitalization Action Plan Delivery

1. PLAN: Define the Aim, Analyze Root Causes, and Formulate Hypotheses

• Problem Statement: A baseline quality audit in a 300-bed hospital reveals that only 18% of pediatric patients admitted for status asthmaticus receive an updated written Asthma Action Plan prior to discharge, and only 22% attend their 30-day post-hospitalization outpatient follow-up visit.
• SMART Aim Statement: "Increase the percentage of pediatric asthma inpatients discharged with an individualized written Asthma Action Plan from 18% to ≥85% within 90 days, and increase 30-day outpatient follow-up attendance from 22% to ≥60% within 6 months."
• Root Cause Analysis (Fishbone/Ishikawa Diagram):
  • Process: Discharge summaries are printed hastily; paper AAPs are stored in a locked supply room.
  • People: Bedside nurses assume hospitalists complete the plan; hospitalists assume the outpatient clinic will do it.
  • Technology: The EHR lacks an integrated, auto-populating AAP template.
• The Change Concept: Develop an automated, smart-form AAP embedded into the hospital discharge order set that auto-populates the child's prescribed medications, and mandate bedside AE-C counseling prior to discharge.

2. DO: Execute the Test of Change on a Small Pilot Scale

• Small-Scale Test: Test the intervention on one inpatient unit (Pediatric Unit 3A) over a 2-week testing window (sample size: n=20 consecutive pediatric asthma admissions).
• Implementation: An AE-C receives an automated EHR notification when an asthma patient is admitted. The educator visits the bedside 24 hours prior to anticipated discharge, generates the digital AAP, conducts a 30-minute self-management teaching session with return inhaler demonstration, prints the finalized AAP, and books the 14-day outpatient clinic follow-up directly in the scheduling module.

3. STUDY: Analyze the Pilot Data, Compare to Predictions, and Identify Surprises

• Quantitative Results: On Unit 3A, written AAP distribution reached 90% (18/20 patients). Direct 14-day follow-up appointment attendance increased to 65% (13/20 patients).
• Qualitative Observations & Friction Points:
  • For weekend discharges, no AE-C was on duty, and discharge AAP distribution dropped back to 25%.
  • Bedside nurses expressed frustration that the EHR template required too many mandatory drop-down clicks.
  • Spanish-speaking families received English-language AAPs when the AE-C was unavailable.

4. ACT: Refine, Standardize, or Expand Based on Findings

• Adapt the Intervention: Work with clinical informatics to streamline the EHR smart-form to three mandatory clicks. Embed auto-translation features generating simultaneous English/Spanish AAPs.
• Standardize and Scale: Train weekend pediatric charge nurses to deliver the simplified AAP when the AE-C is off-duty. Formulate an official hospital clinical policy: no pediatric asthma patient can be discharged without an attached, validated AAP.
• Launch PDSA Cycle 2: Expand the refined protocol hospital-wide to the Pediatric Intensive Care Unit (PICU) and Emergency Department.

---

Reporting Outcomes to Executive Stakeholders and Payers

Clinical asthma educators must translate clinical improvements into executive business language. Hospital C-suites (Chief Executive Officers, Chief Medical Officers, Chief Financial Officers) and commercial health plan directors operate under financial constraints and quality ratings systems.

Return on Investment (ROI) vs. Value on Investment (VOI)

• Return on Investment (ROI): Direct financial cost avoidance divided by program implementation costs: $\text{ROI} = \frac{\text{Net Financial Cost Avoidance} - \text{Program Operational Costs}}{\text{Program Operational Costs}} \times 100$ Clinical Example: An outpatient AE-C clinic costs $150,000 annually (salary, benefits, materials). In its first year, the program enrolls 200 high-risk pediatric asthmatics, reducing annualized ED visits from 250 to 100 (avoiding 150 ED visits at $1,200 each = $180,000 saved) and reducing inpatient admissions from 40 to 12 (avoiding 28 hospitalizations at an average cost of $7,500 each = $210,000 saved). Total direct healthcare cost avoidance = $390,000. $\text{Net Savings} = \$390,000 - \$150,000 = \$240,000$ $\text{ROI} = \frac{\$240,000}{\$150,000} \times 100 = 160\% \quad (\text{Every } \$1.00 \text{ invested yields } \$2.60 \text{ in total return})$
• Value on Investment (VOI): Broad organizational value that extends beyond direct hard-dollar savings. VOI includes enhanced patient and family satisfaction (driving higher Hospital Consumer Assessment of Healthcare Providers and Systems [HCAHPS] scores), reduced caregiver work absenteeism, reduced school absenteeism (which protects public school municipal funding tied to average daily attendance [ADA]), and capacity creation—freeing high-acuity inpatient and PICU beds for surgical or tertiary patients.

Alignment with National Quality Standards: NCQA HEDIS Measures

The National Committee for Quality Assurance (NCQA) establishes the Healthcare Effectiveness Data and Information Set (HEDIS), used by over 90% of America's health plans to measure performance. Two HEDIS measures directly impact asthma care:

1. Asthma Medication Ratio (AMR): Measures the percentage of patients aged 5–64 with persistent asthma who had a ratio of controller medications to total asthma medications (controller plus reliever) of ≥0.50 during the measurement year. An AMR <0.50 indicates that rescue SABA canisters outnumber controller canisters, serving as an independent statistical predictor of imminent severe exacerbations. Health plans reward clinical networks that achieve superior AMR benchmarks.
2. Medication Adherence for Controller Medications (PDC): Evaluates the Proportion of Days Covered (PDC) for individuals prescribed inhaled corticosteroids or combination ICS-LABA, setting a quality threshold of ≥75% adherence across the calendar year.