8.1 Formative vs. Summative Evaluation Methods & Survey Design

Formative vs. Summative Evaluation Methods & Survey Design

Public park and recreation agencies operate as civic institutions entrusted with public funds, facilities, and community well-being. Consequently, programming decisions must never rely on anecdotal impressions, personal assumptions, or historical inertia. Program evaluation is the formal, systematic collection and analysis of data to determine the merit, worth, quality, and effectiveness of recreation services. On the Certified Park and Recreation Professional (CPRP) examination, candidates are tested extensively on their ability to select appropriate evaluation models, design valid survey instruments, interpret quantitative and qualitative metrics, and leverage data to fulfill professional accountability standards such as the Commission for Accreditation of Park and Recreation Agencies (CAPRA).

---

1. The Core Purposes of Recreation Program Evaluation

Evaluation is not an isolated administrative task performed at the end of a fiscal year; it is an integral, continuous component of the recreation programming cycle. Evaluation serves five fundamental administrative purposes:

+-----------------------------------------------------------------------------------+
|                    CORE PURPOSES OF RECREATION PROGRAM EVALUATION                 |
+-----------------------------------------------------------------------------------+
| 1. ACCOUNTABILITY & GOVERNANCE   --> Justify public tax subsidies, grants, and    |
|                                      compliance with CAPRA and municipal boards.  |
| 2. COST-EFFECTIVENESS & FINANCE  --> Determine cost per participant hour, pricing |
|                                      recovery targets, and eliminate fiscal waste.|
| 3. QUALITY IMPROVEMENT (CQI)     --> Identify instructional weaknesses, safety   |
|                                      hazards, and optimize facility scheduling.   |
| 4. PARTICIPANT SATISFACTION      --> Measure customer expectations, Net Promoter  |
|                                      Scores, and foster long-term patron retention.|
| 5. FUTURE STRATEGIC PLANNING     --> Guide seasonal program matrix revisions, CIP |
|                                      priorities, and Master Plan alignment.       |
+-----------------------------------------------------------------------------------+

1. Public Accountability and Stewardship: Demonstrating to elected officials, funding bodies, advisory boards, and taxpayers that public dollars and resources are managed ethically, equitably, and effectively.
2. Cost-Effectiveness and Resource Optimization: Analyzing whether the program delivers sufficient community value relative to its operational costs, identifying opportunities to reallocate staff or subsidy funding to higher-need areas.
3. Continuous Quality Improvement (CQI): Pinpointing operational bottlenecks, instructional deficiencies, facility maintenance needs, or safety risks to refine program delivery.
4. Participant Satisfaction and Experience Quality: Assessing whether participants' expectations were met or exceeded, measuring perceived value, and building customer loyalty across diverse demographics.
5. Evidence-Based Strategic Planning: Providing the empirical foundation for seasonal program brochure development, comprehensive master plans, capital improvement planning, and departmental accreditation reviews.

---

2. Formative vs. Summative Evaluation

The two primary classifications of program evaluation are Formative Evaluation and Summative Evaluation. Understanding the distinction between process-oriented and outcome-oriented assessment is critical for CPRP exam scenarios.

+-----------------------------------------------------------------------------------+
|                     FORMATIVE VS. SUMMATIVE EVALUATION                            |
+-----------------------------------------------------------------------------------+
| DIMENSION          | FORMATIVE EVALUATION           | SUMMATIVE EVALUATION        |
|--------------------|--------------------------------|-----------------------------|
| Primary Focus      | Process, operations, delivery  | Outcomes, impact, value     |
| Timing             | Concurrent / Mid-program       | Terminal / Post-program     |
| Core Purpose       | Immediate mid-course correction| Final judgment of success   |
| Key Question       | "How can we improve it NOW?"   | "Did we achieve the goals?" |
| Primary Beneficiary| Current enrollees & instructors| Future cohorts, leadership  |
| Typical Tools      | Weekly check-ins, site audits  | Exit surveys, pre/post tests|
+-----------------------------------------------------------------------------------+

A. Formative Evaluation (Process-Oriented / Ongoing)

• Definition: Evaluation conducted during the planning and implementation phases of a recreation program to monitor ongoing operational delivery.
• Primary Objective: Provide immediate, actionable feedback to instructors, coordinators, and supervisors so that necessary mid-course adjustments can be made before the program concludes.
• Practical Applications:
  • A summer day camp supervisor conducts weekly Wednesday check-ins with counselors to modify field trip pacing and adjust water hydration schedules during an unexpected heat wave.
  • An adult fitness instructor uses a 2-minute thumbs-up/thumbs-down check at the end of Week 2 of an 8-week bootcamp to calibrate exercise intensity for beginner participants.
  • An aquatics coordinator performs an unannounced lifeguard positioning and scanning audit during Week 3 of swimming lessons to correct blind spots immediately.
• Analogy: "When the chef tastes the soup while cooking, that is formative evaluation; when the guests taste the soup, that is summative evaluation." (Robert E. Stake).

B. Summative Evaluation (Outcome-Oriented / Concluding)

• Definition: Evaluation conducted at the conclusion of a program or program cycle to assess overall performance, goal attainment, participant outcomes, and final impact.
• Primary Objective: Make high-level administrative judgments regarding whether the program achieved its stated SMART objectives, met cost-recovery targets, and should be continued, expanded, redesigned, or discontinued.
• Practical Applications:
  • Administering an end-of-season digital survey to youth basketball parents evaluating coach sportsmanship, referee fairness, and facility lighting.
  • Analyzing pre- and post-test scores from a 6-week arthritis aquatic therapy program to verify a statistically significant increase in joint mobility.
  • Conducting an end-of-year cost-recovery analysis comparing direct operating expenditures against total registration fee revenue to determine if the class met its 100% direct cost-recovery mandate.

---

3. Quantitative vs. Qualitative Evaluation Instruments

A robust evaluation framework incorporates triangulation—the strategic combination of both quantitative and qualitative methods to gain a comprehensive, objective understanding of program performance.

A. Quantitative Evaluation Methods (Numerical & Generalizable)

Quantitative methods collect objective, measurable numerical data that can be statistically aggregated, compared across time, and generalized across large populations.

1. Likert-Scale Survey Questionnaires:
   • Rating statements utilizing standardized ordinal scales (e.g., 1 = Strongly Disagree to 5 = Strongly Agree, or 1 = Poor to 5 = Excellent).
   • Measures participant perceptions of instructor knowledge, facility cleanliness, safety, and overall value.
2. Attendance Headcounts and Retention Rates:
   • Tracking initial registration capacity, daily drop-in counts, weekly attendance drop-off rates, and retention across sequential program sessions.
3. Pre-Test / Post-Test Objective Skill Assessments:
   • Standardized, criterion-referenced skill testing administered before the first session and immediately following the final session (e.g., American Red Cross swim stroke competency rubrics, youth archery target scoring).
4. Cost per Participant Hour (CPPH):
   • A vital financial quantitative metric: Total Direct Operating Costs divided by total aggregate participant contact hours.

B. Qualitative Evaluation Methods (Descriptive & Contextual)

Qualitative methods gather rich, narrative, descriptive data that reveals the underlying motivations, emotional responses, cultural contexts, and subtle nuances of the participant experience.

1. Focus Groups:
   • Structured discussions with 6 to 10 pre-screened participants, facilitated by a trained moderator using standardized, open-ended question prompts.
   • Uncovers deep personal feedback regarding programmatic barriers, cultural relevance, and perceived community benefits.
2. Participant and Parent Exit Interviews:
   • Semi-structured, one-on-one intercept conversations conducted immediately following an activity or special event.
3. Instructor and Staff Debriefing Logs:
   • Written narrative logs completed by instructors after each class session, detailing behavioral challenges, equipment malfunctions, facility temperature issues, or curriculum pacing challenges.
4. Unobtrusive Direct Observation:
   • Systematic observational rubrics where an evaluator observes participant engagement, social interactions, inclusion behaviors, and facility flow without interfering in the activity.

---

4. Survey Design Best Practices & Methodological Pitfalls

Surveys represent the most widely used evaluation tool in municipal parks and recreation. However, a poorly designed survey generates skewed, invalid data that misleads administrative decision-making. The CPRP candidate must master survey construction principles.

+-----------------------------------------------------------------------------------+
|                         SURVEY DESIGN PITFALLS TO ELIMINATE                       |
+-----------------------------------------------------------------------------------+
| 1. LEADING QUESTIONS          --> "How much did you love our award-winning park?" |
|    Corrected Wording          --> "How satisfied were you with the park condition?|
|-------------------------------|---------------------------------------------------|
| 2. DOUBLE-BARRELED QUESTIONS  --> "Was the instructor friendly and knowledgeable?"|
|    Corrected Wording          --> Split into two items: 1) Friendliness, 2) Knowl.|
|-------------------------------|---------------------------------------------------|
| 3. UNBALANCED SCALES          --> [Excellent, Very Good, Good, Fair] (No Negative)|
|    Corrected Wording          --> [Very Satisfied, Satisfied, Neutral, Dissat...] |
|-------------------------------|---------------------------------------------------|
| 4. JARGON / COMPLEX READING   --> "Evaluate the department's CIP amortization."  |
|    Corrected Wording          --> Use plain 6th-8th grade language accessible to all|
+-----------------------------------------------------------------------------------+

Key Principles of Valid Survey Construction

• Avoid Leading and Loaded Questions: Questions must maintain strict neutrality. Never prompt the respondent toward a favored answer.
  • Flawed: "Do you agree that the city should expand our highly successful youth soccer program?"
  • Valid: "To what extent do you support or oppose expanding youth soccer program capacity?"
• Eliminate Double-Barreled Questions: A double-barreled question combines two separate inquiries into a single statement, making it impossible to determine which element the respondent is evaluating.
  • Flawed: "The recreation center was clean and well-staffed." (A patron might find the center spotless but severely understaffed).
  • Valid: Evaluate cleanliness and staffing adequacy as two distinct survey items.
• Use Balanced Response Scales with Clear Anchors: Ensure equal positive and negative options around an explicit neutral midpoint (e.g., 5-point Likert scale: 1 = Very Dissatisfied, 2 = Dissatisfied, 3 = Neutral/Neither, 4 = Satisfied, 5 = Very Satisfied).
• Accessibility and Universal Design:
  • Adhere to ADA Title II communication requirements by offering large-print text, screen-reader-compatible digital web forms, and multi-language translations reflective of community demographics (e.g., Spanish, Vietnamese, Mandarin).
  • Keep reading levels at the 6th to 8th-grade level to ensure broad comprehension.
• Survey Length and Fatigue Mitigation: Limit survey duration to 3 to 7 minutes (10 to 15 concise questions). Lengthy questionnaires induce respondent fatigue, resulting in dropped completion rates and erratic responses.

---

5. Formative vs. Summative Evaluation Comparison Table