11.2 Program Evaluation Methods & Quality Improvement

Program Evaluation Methods & Quality Improvement

Core Clinical Mandate: While client evaluation measures individual functional progress, program evaluation systematically examines the effectiveness, efficiency, safety, and clinical value of the entire therapeutic recreation service delivery system. In modern value-based healthcare, a Certified Therapeutic Recreation Specialist (CTRS) must not only be an expert clinician at the bedside, but also an active contributor to Continuous Quality Improvement (CQI). Utilizing validated quality frameworks—such as Avedis Donabedian's Structure-Process-Outcome model and the Plan-Do-Study-Act (PDSA) cycle—ensures that RT programs consistently deliver high-quality, cost-effective, evidence-based care that satisfies regulatory standards and enhances patient outcomes.

---

Client Evaluation vs. Program Evaluation

Understanding the distinction between micro-level (client) and macro-level (program) evaluation is essential for CTRS administrative and clinical competency:

• Client-Level Evaluation (Micro): Focuses exclusively on the individual patient's functional gains, behavioral change, and goal attainment (e.g., "Did Client X achieve independent wheelchair transfers during community outings?").
• Program-Level Evaluation (Macro): Aggregates data across entire client cohorts, diagnostic groups, or departmental operations to assess service efficacy, resource utilization, protocol adherence, cost-effectiveness, and consumer satisfaction (e.g., "Does our 6-week Spinal Cord Injury Community Re-entry Program systematically reduce 30-day post-discharge hospital readmissions and improve overall community participation scores?").

+-------------------------------------------------------------------------------------------------+
|                      THE QUALITY IMPROVEMENT CONTINUUM IN RECREATIONAL THERAPY                  |
|                                                                                                 |
|   +-----------------------------------------------------------------------------------------+   |
|   |                     DONABEDIAN'S HEALTHCARE QUALITY TRIAD                               |   |
|   |                                                                                         |   |
|   |   +-------------------+      +-------------------+      +---------------------------+   |   |
|   |   |     STRUCTURE     | ---> |      PROCESS      | ---> |          OUTCOME          |   |   |
|   |   | Resources, Staff, |      | Clinical Delivery,|      | Functional Recovery, LOS, |   |   |
|   |   | Facilities, Budget|      | Protocol Adherence|      | Patient Satisfaction, QOL |   |   |
|   |   +-------------------+      +-------------------+      +---------------------------+   |   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                |                                                |
|                                                v                                                |
|   +-----------------------------------------------------------------------------------------+   |
|   |                     CQI / TQM ITERATIVE ENGINE: THE PDSA CYCLE                          |   |
|   |                                                                                         |   |
|   |         +------------+                       +------------+                             |   |
|   |         |    PLAN    | --------------------> |     DO     |                             |   |
|   |         | Root Cause |                       | Small-Scale|                             |   |
|   |         | AIM & Plan |                       | Pilot Test |                             |   |
|   |         +------------+                       +------------+                             |   |
|   |               ^                                     |                                   |   |
|   |               |                                     v                                   |   |
|   |         +------------+                       +------------+                             |   |
|   |         |    ACT     | <-------------------- |   STUDY    |                             |   |
|   |         | Standardize|                       | Analyze &  |                             |   |
|   |         | Adopt/Adapt|                       | Benchmark  |                             |   |
|   |         +------------+                       +------------+                             |   |
|   +-----------------------------------------------------------------------------------------+   |
+-------------------------------------------------------------------------------------------------+

---

Avedis Donabedian's Quality Framework

Physician and healthcare researcher Avedis Donabedian established the most universally recognized model for assessing healthcare quality (1966). Donabedian proposed that healthcare quality must be evaluated across three interdependent dimensions: Structure, Process, and Outcome.

1. Structure Indicators

Structure refers to the physical, organizational, financial, and human resource attributes of the setting in which healthcare is delivered. High-quality structure is the prerequisite foundation for effective clinical processes.

• Physical Facilities & Environment: Accessibility of therapeutic recreation clinic spaces, outdoor challenge courses, sensory integration rooms (Snoezelen), aquatic therapy pools, and adaptive sports fields complying with ADA accessibility guidelines.
• Equipment & Supplies: Availability, safety inspections, and sanitization of adaptive equipment (handcycles, sport wheelchairs, adaptive fishing gear, biofeedback units, musical instruments).
• Human Resources & Credentials: Percentage of recreation staff holding active NCTRC CTRS certification, staff-to-patient staffing ratios, state licensure credentials, specialty certifications (e.g., CBIS, ATP, CARSS), and verified annual continuing education units (CEUs).
• Fiscal & Operational Structure: Departmental operating budgets, capital equipment replacement funds, and comprehensive departmental Policy and Procedure (P&P) manuals.

2. Process Indicators

Process refers to the actual delivery of healthcare services—what is done to and for the patient, and how skillfully clinical interventions are executed. Process measures evaluate whether clinical care adheres to established professional standards and clinical practice guidelines.

• Timeliness of the APIED Cycle: Percentage of initial assessments completed within regulatory timeframes (e.g., within 24 to 72 hours of admission); percentage of Individualized Treatment Plans (ITPs) finalized and signed prior to intervention delivery.
• Protocol Adherence and Clinical Fidelity: Degree to which CTRS staff deliver interventions according to validated evidence-based clinical protocols (e.g., protocol fidelity in a 4-week Dialectical Behavior Therapy-informed RT coping group).
• Documentation Quality & Compliance: Completeness of daily/weekly SOAP progress notes, clear objective functional goal tracking, and explicit physician order verification.
• Facilitation Technique Competence: Observational peer audits evaluating the therapist's use of active listening, de-escalation strategies, biomechanical transfer safety, and prompt fading.

3. Outcome Indicators

Outcome refers to the effects of healthcare interventions on the health status, functional abilities, quality of life, and satisfaction of patients and populations. In value-based care, outcomes provide the ultimate justification for service delivery.

• Functional Independence Gains: Quantified improvement on standardized functional instruments (e.g., delta scores on CMS Section GG, CERT-Phys Rehab, CERT-Psych, or LCM).
• Clinical & Institutional Metrics: Reduction in acute hospital Length of Stay (LOS), reduction in 30-day post-discharge emergency room readmissions, decrease in behavioral code events, and reduction in chemical or physical restraint utilization.
• Patient & Family Satisfaction: Standardized ratings on Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS), Press Ganey surveys, and RT-specific satisfaction questionnaires.
• Long-Term Community Re-Integration: Percentage of clients maintaining independent community recreation participation, return to competitive or supported employment, and reported self-efficacy at 3-month and 6-month post-discharge follow-ups.

---

Donabedian Quality Framework Matrix in RT Practice

---

Continuous Quality Improvement (CQI) and TQM Methodologies

Continuous Quality Improvement (CQI) and Total Quality Management (TQM) represent proactive, organizational philosophies originating from W. Edwards Deming and Joseph Juran. In contrast to traditional Quality Assurance (QA)—which focused reactively on inspecting defects and punishing individual clinicians—CQI operates on the core principle that 85% of clinical errors and inefficiencies arise from flawed processes and systems, not individual incompetence.

Core CQI Principles in Healthcare RT

1. Customer-Driven Focus: Identifying both internal customers (interdisciplinary team members, nurses, physicians) and external customers (patients, families, payers, accreditation agencies) and aligning services with their needs.
2. Data-Driven Decision Making: Utilizing objective quantitative data (run charts, histograms, control charts, Pareto diagrams) rather than subjective clinical impressions to identify problems and measure improvements.
3. Non-Punitive Culture of Safety: Encouraging staff to report near-misses, process bottlenecks, and clinical concerns without fear of disciplinary retaliation.
4. Systematic Root Cause Analysis: Drilling down to systemic vulnerabilities rather than applying superficial, temporary fixes.

---

The Plan-Do-Study-Act (PDSA) Improvement Cycle

The Plan-Do-Study-Act (PDSA) cycle (also known as the Deming Wheel or Shewhart Cycle) is the primary iterative scientific framework utilized in healthcare quality improvement.

+-------------------------------------------------------------------------------------------------+
|                        THE PDSA QUALITY IMPROVEMENT CYCLE IN RT                                 |
|                                                                                                 |
|   [ PLAN ]                                               [ DO ]                                 |
|   - Identify problem: 42% of assessments late            - Pilot standardized electronic intake |
|   - Conduct 5 Whys & Fishbone root cause analysis        - Test on Unit 3B for 30-day trial     |
|   - Draft SMART aim: Increase on-time to >90%            - Deliver 1-hour staff training        |
|   - Formulate change intervention & metrics              - Collect qualitative & process data   |
|                      |                                              |                           |
|                      v                                              v                           |
|   [ ACT ]                                                [ STUDY ]                              |
|   - ADOPT: Standardize across all 4 hospital units       - Analyze post-pilot on-time rate      |
|   - Update Departmental Policy & Procedure Manual        - Result: 94% on-time completion       |
|   - Institutionalize annual EHR competency check         - Identify remaining bottleneck        |
|   - Plan next PDSA cycle for treatment plans             - Document unintended staff burdens    |
+-------------------------------------------------------------------------------------------------+

Operational Stages of the PDSA Cycle

1. Plan:
   • Problem Identification: Define the clinical or operational deficit (e.g., "Inpatient behavioral health clients are experiencing a 35% absenteeism rate in morning leisure education groups").
   • Root Cause Analysis (RCA): Deploy analytical tools such as the Fishbone (Ishikawa) Diagram (categorizing causes by People, Methods, Machines, Materials, Environment) or the 5 Whys Technique to uncover underlying systemic drivers (e.g., morning medication administration schedules conflict directly with the 09:00 group start time).
   • SMART Aim Statement: Formulate an explicit goal: "Increase morning RT group attendance from 65% to $\ge 90%$ within 60 days on the adult psychiatric unit."
   • Change Strategy: Reschedule the leisure education group to 10:00 AM, following medication rounds, and implement a visual daily schedule board on the unit.
2. Do:
   • Execute the change on a small-scale pilot basis (e.g., on Unit 2A for a 4-week trial).
   • Collect continuous process and outcome data; document unexpected hurdles, staff feedback, and client responses.
3. Study (Check):
   • Analyze the collected data against baseline metrics.
   • Compare actual results to the projected aim (e.g., attendance increased to 92%, confirming the hypothesis).
   • Evaluate unintended consequences (e.g., did shifting the time create a conflict with occupational therapy gym times?).
4. Act:
   • Adopt: If the pilot successfully achieved the aim without negative side effects, standardize the new group schedule across all psychiatric units, update the departmental Policy and Procedure manual, and educate all multidisciplinary staff.
   • Adapt: If the pilot showed partial success (e.g., attendance rose to 78%), refine the intervention (e.g., add a peer-reminder system) and launch a second iterative PDSA cycle.
   • Abandon: If the pilot failed or created severe operational conflicts, discard the change and formulate an alternative hypothesis.

---

Lean and Six Sigma Methodologies in RT Management

Modern healthcare organizations combine CQI with Lean and Six Sigma methodologies to eliminate waste and reduce clinical process variability.

1. Lean Methodology (Eliminating Waste)

Originating from the Toyota Production System, Lean focuses on maximizing customer value while eliminating muda (waste). In recreational therapy departments, Lean targets the 8 classic forms of healthcare waste:

• Waiting: Clients waiting in hallways for transport to RT sessions; therapists waiting for physician order signatures.
• Transportation: Inefficient routing of clients across hospital wings for recreation therapy groups.
• Over-Processing: Documenting identical functional observations in multiple redundant sections of the Electronic Health Record.
• Defects / Rework: Incomplete intake assessment forms requiring the CTRS to re-interview patients for missing data.
• Underutilized Talent: Assigning CTRS clinicians to perform clerical scheduling or portering tasks instead of skilled clinical therapy.

2. Six Sigma (Reducing Process Variation)

Six Sigma utilizes the DMAIC framework (Define, Measure, Analyze, Improve, Control) to eliminate defects and achieve near-perfection (fewer than 3.4 defects per million opportunities). In RT, Six Sigma is utilized to eliminate medication timing errors during community outings, standardize assessment scoring reliability among therapists, and eliminate billing code rejections.

---

Utilization Review (UR)

Utilization Review (UR) is a formal, systematic evaluation of the medical necessity, appropriateness, efficiency, and efficacy of healthcare services, procedures, and facilities. UR ensures that healthcare resources are allocated judiciously, preventing both under-utilization (denying necessary therapy) and over-utilization (providing unnecessary or ineffective care).

The Three Phases of Utilization Review

+-------------------------------------------------------------------------------------------------+
|                            THE THREE PHASES OF UTILIZATION REVIEW                               |
|                                                                                                 |
|   +--------------------------+  +--------------------------+  +------------------------------+  |
|   |    PROSPECTIVE REVIEW    |  |    CONCURRENT REVIEW     |  |     RETROSPECTIVE REVIEW     |  |
|   |     (Pre-Admission /     |  |   (Active Inpatient /    |  |       (Post-Discharge /      |  |
|   |    Prior Authorization)  |  |    Continued Stay)       |  |      Payer Billing Audit)    |  |
|   |                          |  |                          |  |                              |  |
|   | - Verifies diagnostic    |  | - Evaluates ongoing      |  | - Reviews closed medical     |  |
|   |   criteria for admission |  |   functional progress    |  |   records for compliance     |  |
|   | - Justifies RT service   |  | - Justifies continued    |  | - Verifies billed codes match|  |
|   |   intensity & dosage     |  |   skilled stay vs plateaul  |   documented skilled care    |  |
|   | - Obtains payer prior    |  | - Authorizes additional  |  | - Identifies systemic denial |  |
|   |   approval for therapy   |  |   approved days          |  |   patterns to appeal/remedy  |  |
|   +--------------------------+  +--------------------------+  +------------------------------+  |
+-------------------------------------------------------------------------------------------------+

Defending Medical Necessity in RT Documentation

To withstand utilization review audits and prevent third-party payer denials, the CTRS must explicitly document:

1. Skilled Care Requirement: Why the specialized clinical expertise, assessment, and judgment of a CTRS are mandatory, proving the intervention cannot be safely or effectively provided by non-certified recreation staff, aides, or volunteers.
2. Active Remedial Treatment: Clear linkage to restorative goals aimed at improving functional status, distinct from maintenance or diversional recreation.
3. Quantifiable Trajectory: Objective data proving ongoing progress toward measurable functional benchmarks.

---

Risk Management Audits, Incident Analysis & Client Satisfaction

1. Clinical Incident Analysis & Root Cause Analysis (RCA)

When an adverse clinical event occurs (e.g., client fall during an adaptive cycling session, elopement during a community outing, contraband introduction in a psychiatric art group), risk management protocols mandate:

• Immediate Medical & Safety Stabilization: Providing first aid, securing the environment, and notifying the attending physician.
• Objective Incident Reporting: Filing a factual, non-judgmental incident report within 24 hours. Critical rule: Incident reports are internal risk management documents and are never referenced or filed directly within the client's official medical record.
• Root Cause Analysis (RCA): Conducting a multidisciplinary, retrospective deep-dive to identify underlying latent systemic vulnerabilities (e.g., equipment maintenance lapse, inadequate staffing ratio during high-risk transfer, ambiguous policy regarding community outing screening).

2. Client and Family Satisfaction Surveys

Client satisfaction is a direct indicator of service responsiveness, person-centered dignity, and program quality. Best practices for RT satisfaction surveying include:

• Administering surveys at discharge or immediately post-program cycle using standardized 5-point Likert scales.
• Surveying across key dimensions: perceived therapeutic benefit, staff professionalism and empathy, accessibility of facilities/equipment, and empowerment in leisure choice.
• Systematically aggregating quantitative satisfaction scores and analyzing qualitative narrative comments during monthly departmental CQI committee meetings.