14.2 Validation and Benefits Realization

14.2 Validation and Benefits Realization

Quick Answer: Task D.3 checks the live system against the contract and the design specification. Task D.4 checks whether expected benefits showed up—ROI, benchmarks, and user satisfaction—against a baseline you captured before go-live. A cutover party is neither validation nor realization.

D.1 and D.2 asked whether you tested under control. D.3 and D.4 ask a colder pair of questions: Did we get what we signed? and Did it help? Domain 3 is 30% of CPHIMS because this is where capital, safety, and reputation either convert or evaporate. Chapter 11 built the cost-benefit case and the proposal; Chapter 13 implemented inside scope, schedule, budget, and quality. This section is the after-action that those documents promised.

CPHIMS practice questionsPractice questions with detailed explanations

Validation is not another word for testing

Testing (14.1) asks whether functions behave. Validation (D.3) asks whether the implemented system matches two external authorities:

1. Contractual terms — statement of work, exhibit lists, interface schedule, training hours, uptime or response SLAs, acceptance criteria, data-return/destroy clauses, and any BAA duties that were sold as part of the deal.
2. Design specifications — approved requirements, workflow and technical design, identity and role model, report catalog, downtime path, and the test plan’s must-pass set.

You can have green UAT and still fail D.3. Example: testers love the new portal, but the SOW promised a bidirectional medication history feed and a 15-minute RTO downtime viewer that were never built. Example: the design specified two-identifier transfusion checks; the live build allows a single identifier after a “temporary” waiver nobody closed.

Contract validation is a line-by-line read, not a feeling. Pull the SOW and the RFP response (Chapter 11.3). For each deliverable, record: present as specified, present with a named variance, or missing. Variances need a change order or a credit—not a shrug. SLA validation uses measured uptime, queue age, and ticket age against the contracted definition, including the exclusions the vendor wrote into the footnotes. If the SLA says 99.9% but excludes maintenance, third-party networks, and “events beyond control,” say so in the validation file; do not report the headline percentage as if it were earned.

Design validation traces must-have requirements to the live configuration. Use the same MoSCoW language as analysis (Chapter 10.2): Musts that are absent are defects, not enhancements. Safety, identity, and downtime continuity remain Musts after go-live. A design spec that said “break-the-glass with extra audit” is not validated by a standing God-mode pool created “so UAT would go faster.”

Formal acceptance is the control that binds D.1’s acceptance test to D.3. A signed acceptance (or a signed punch list with dates and owners) is how you later dispute a vendor invoice or release retainage. “We went live, so we accepted” is how organizations pay for software they never received.

Benefits realization is a measurement program

Task D.4: evaluate that expected benefits are achieved and report metrics. HIMSS names the examples: return on investment, benchmarks, and user satisfaction. This is not the pre-decision cost-benefit analysis (task A.8). That analysis forecast. Realization measures.

You cannot realize a benefit you never defined. The proposal (task A.9) should have named the benefit, the metric, the baseline, the target, the owner, and the review dates. If it did not, write that packet now—honestly—and do not invent a victory.

Return on investment (ROI) in CPHIMS language is realized benefit relative to realized cost over a stated period. Keep the arithmetic humble and the definitions loud:

• Costs include license, implementation, interfaces, devices, training time, overtime, lost productivity during hypercare, and ongoing FTE. Hidden costs that were “someone else’s budget” still count if you claim a system-level ROI.
• Benefits must be attributable and measured: reduced duplicate testing, shorter registration time, fewer status calls, avoided locums, captured charges, reduced harm events. Do not double-count the same dollar in quality and finance.
• Net benefit (benefits minus costs) is often clearer than a ratio when the time horizon is short.
• Report projected versus realized in one table. A 200% slide-deck ROI that becomes a 20% realized ROI is a finding, not a scandal, if you learn from it. Calling the projection “actual” is the scandal.

Benchmarks compare your metric to a baseline (yourself last year, or the month before go-live) and, when honest peers exist, to an external reference (specialty society, collaborative, or internal system quartile). A benchmark without a baseline is a number looking for a story. Do not invent a HIMSS or CMS “national average” on the exam; the skill is the comparison design, not a memorized percentile. Clinical benchmarks (sepsis bundle, med-scan rate, result turnaround) and operational benchmarks (registration minutes, claim-denial rate, inbox age) both count if they were in the expected-benefit list.

User satisfaction is a designed instrument: a short, repeated survey of the people who do the work; structured interviews; and operational proxies such as ticket volume, after-hours pages, workaround audits, and voluntary adoption of an optional path. Satisfaction is not “the CMIO is happy,” “training attendance was high,” or “login counts went up.” High logins with high workarounds is a failed benefit. Pair satisfaction with effectiveness: can the nurse complete med pass faster and more safely, or only click more?

When and how you report

Set the clock before go-live:

• Hypercare (first 1–2 weeks): defect burn-down, safety signals, command-center themes. This is stabilization, not ROI.
• 30 and 90 days: first look at process metrics, satisfaction pulse, SLA performance, punch-list closure. Early ROI is usually incomplete; say so.
• 6–12 months: the realization review the board was promised—costs actually incurred, benefits actually moved, remaining gaps, and a keep/fix/sunset recommendation.

Owners present the report. Informatics can assemble the file; the operational sponsor owns whether the benefit landed. Finance owns the cost side. Quality owns clinical outcome claims. If those seats are empty, you have a governance problem, not a dashboard problem.

Tie realization back to validation. A benefit that required a missing interface cannot appear. Do not “train harder” a design that failed D.3.

Scenarios and exam traps

Scenario. The vendor invoices the final milestone because “production is live.” The SOW still lists a claims attachment interface and 40 hours of at-the-elbow training that were cancelled to hit the date. D.3 says withhold or change-order; do not confuse a cutover with contractual acceptance.

Scenario. Leadership reports a 3× ROI six weeks after an ambulatory EHR optimization because “we believed the business case.” No baseline cycle time was captured, and overtime during training was omitted from costs. D.4 fails. Publish projected versus realized with a complete cost list, or decline to claim ROI yet.

Scenario. Inbox volume dropped, but a satisfaction pulse shows nurses created a shadow spreadsheet because the new in-basket is unusable. Login counts look fine. Report the spreadsheet as a failed benefit and reopen design; do not celebrate adoption.

Scenario. Peer hospitals quote a “top-decile scan rate.” You have no pre-go-live scan rate of your own. Start with the internal baseline. External benchmarks without a local denominator are decoration.

Watch these traps:

1. Treating go-live, an invoice, or a party as D.3 validation.
2. Confusing the Chapter 11 forecast with realized ROI.
3. Claiming benefits without a baseline or with incomplete costs.
4. Using sponsor mood or login counts as user satisfaction.
5. Ignoring SLA exclusions and claim windows when “validating” uptime.
6. Training your way around a missing contracted deliverable.

When D.3 and D.4 are closed honestly, the organization knows what it owns and whether it was worth it. Domain 3 then turns to the standing duty that never closes: privacy and security.