15.2 Program Evaluation, Agency Evaluation Methods, and Quality Assurance

If Section 15.1 is about whether the intervention is working for one client, this section is about whether the program is working for the population it serves. The ASWB Clinical exam treats agency-level evaluation as a core intervention competency because clinical social workers — especially in agencies — are expected to participate in, lead, and ethically use program evaluation and quality assurance activities.

Why Program Evaluation Matters for the Exam

The exam commonly presents a vignette in which an agency receives a grant, a funder requires outcomes data, or a program appears to be underperforming, and asks the social worker to identify the appropriate evaluation approach, the next step in a logic model, or the right quality-improvement method. Knowing the vocabulary — formative, summative, process, outcome, cost-effectiveness, cost-benefit, CQI, PDSA — is the difference between recognizing the correct answer and guessing.

Types of Program Evaluation

Program evaluation is the systematic assessment of a program's design, implementation, or outcomes. The major types answer different questions and are not interchangeable.

Process evaluation asks whether the program is being delivered as designed (fidelity, dosage, reach). Outcome evaluation asks whether the program produced the intended change. A program can be delivered faithfully yet produce no outcomes; it can also produce apparent outcomes that are an artifact of low fidelity. Both evaluations are needed.

The Logic Model

A logic model is the standard scaffold that links resources to results. It is required by most federal and foundation funders and is the single most tested program-evaluation concept on the exam.

Outcomes are usually subdivided into short-term (knowledge, attitudes), intermediate (behavior), and long-term (population-level). Confusing outputs (counts) with outcomes (changes) is a classic exam error.

Diagram: The Logic Model Flow

flowchart LR
  Inputs[Inputs: funding, staff, training] --> Activities[Activities: counseling sessions, consultation]
  Activities --> Outputs[Outputs: 320 sessions, 48 students served]
  Outputs --> ShortOutcomes[Short-term outcomes: improved coping, better attendance]
  ShortOutcomes --> IntOutcomes[Intermediate outcomes: fewer disciplinary referrals, higher grades]
  IntOutcomes --> Impact[Impact: reduced dropout rate over 3 years]
  Feedback[Feedback loop: evaluation data] -.-> Inputs
  Feedback -.-> Activities

The dotted feedback line is the evaluation loop: data from outcomes and impact feed back into inputs and activities, which is how a program improves over time.

Quantitative and Qualitative Evaluation

Program evaluation can be quantitative (e.g., pre/post scores, comparison-group designs, counts of services), qualitative (e.g., participant interviews, focus groups, ethnographic observation), or mixed-methods. Qualitative methods are particularly suited to formative evaluation because they answer why and how a program is or is not working; quantitative methods are suited to summative evaluation because they answer whether and how much. The strongest program evaluations use both.

Quality Assurance and Continuous Quality Improvement

Quality assurance (QA) is the systematic set of activities designed to ensure that services meet defined standards. In social service agencies, QA activities include:

• Program reviews — periodic structured review of program operations against standards
• Audits — financial and clinical record audits, often required by funders
• Peer review — clinicians review one another's cases, typically focusing on quality and appropriateness of care
• Utilization review — verifying that the level and intensity of service match the client's need (especially in managed care)
• Continuous Quality Improvement (CQI) — an ongoing, organization-wide approach that treats quality as something to be built in, not inspected in

The PDSA cycle (Plan-Do-Study-Act) is the engine of CQI:

flowchart LR
  P[Plan: identify a problem and a change to test] --> D[Do: test the change on a small scale]
  D --> S[Study: analyze the results against predictions]
  S --> A[Act: adopt, adapt, or abandon; then start the next cycle]
  A --> P

A key exam point: PDSA is iterative and small-scale. You test a change on a small scale, study what happened, and decide whether to adopt, adapt, or abandon before scaling up. This contrasts with a one-time program audit, which is a single quality-assurance event, not continuous improvement.

Accreditation

Accreditation bodies — such as the Council on Accreditation (COA), the Joint Commission (for behavioral health organizations), and CARF International — set standards that agencies must meet to receive or maintain accreditation. Accreditation typically requires documented policies, measurable outcomes, a current program evaluation plan, a quality improvement process, and evidence of client rights protections. Accreditation is both a quality milestone and, often, a condition of reimbursement or funding.

The Social Worker's Role in Agency Evaluation

Clinical social workers in agencies are not merely passive subjects of evaluation. The role includes:

• Collecting outcome data as part of routine practice (and not back-filling at audit time)
• Participating in peer review as both reviewer and reviewed
• Using evaluation findings to adjust one's own caseload and practice
• Contributing to logic model development because front-line clinicians know what activities and outputs are realistic
• Protecting clients from evaluation activities that could cause harm (e.g., re-traumatizing interviews)

Ethical Use of Evaluation Data

Program evaluation creates strong incentives to present favorable results. The NASW Code of Ethics (1.04 Competence; 1.07 Privacy and Confidentiality; 5.02 Evaluation and Research) requires that social workers conduct evaluation with integrity, obtain informed consent, protect confidentiality of evaluation data, and report findings — including unfavorable ones — honestly. Distorting data to protect a program's funding, omitting negative findings from a funder report, or using client data without consent are all ethical violations. When evaluation results are unfavorable, the ethical response is to use them for improvement (CQI), not to suppress them.