8.3 Predictive Models in Care and Operations

8.3 Predictive Models in Care and Operations

Quick Answer: Descriptive says what happened, diagnostic says why, predictive estimates what may happen, and prescriptive recommends or triggers an action. A model is a governed tool with an intended use. A high AUC is not safe deployment. Do not treat AI as magic, and do not invent unpublished performance numbers to win a meeting.

Domain 2 A.5 pairs interpretation of outcomes with analytics tools. Sections 8.1 and 8.2 were about reports, visuals, and reading a rate. This section is about models that score the future—and about not confusing them with the four analytic types, with CDS, or with a miracle.

/practice/cphimsPractice questions with detailed explanations

Four analytic types — the question, not the logo

A single BI portal can host all four types. Do not let a vendor rename everything “AI.”

A weekly readmission rate is descriptive. A drill-down into missing follow-up is diagnostic. A score that estimates next week’s risk for today’s discharges is predictive. A rule that assigns a transition nurse when that score is high is prescriptive. Those are four requests. They are not “all predictive because the dashboard is modern.”

Predictive is not automatically machine learning. A simple, governed score (age plus a few conditions plus prior utilization) can be predictive. A neural network can be descriptive if all it does is cluster last year’s notes. Judge the question and the intended action.

Prescriptive without capacity is theater. If every “high risk” discharge generates a task and there are three care managers for 80 flags a day, you have built alert fatigue, not a program. Pair the model with a designed workflow, an owner, and a balancing measure (time-to-task, override rate, missed true highs).

CDS versus analytics. A deterioration score that interrupts in the EHR at the moment of a new lactate is acting as clinical decision support (chapter 4 and chapter 9). The same score on a next-morning quality dashboard is analytics. Point-of-care use has a higher safety bar: wrong score, wrong time, wrong patient is a clinical event, not a slide error.

What “good” means — discrimination is not the whole job

When a vendor or data-science team says a model is “accurate,” unpack the word.

Discrimination (often discussed as AUC or similar ranking metrics) asks: if you pick a random patient who had the event and a random patient who did not, how often does the model score the event patient higher? That is a ranking story on a stated sample. It does not say the score is a probability. It does not say the threshold you picked is tolerable. It does not say the model is fair. It does not say the page will fire at a volume nurses can answer.

Calibration asks: among people scored around 20%, did about 20% have the event? A model can rank well and still be miscalibrated in the elderly, in obstetrics, or after a documentation change. Deployment that treats a miscalibrated 0.30 as “30% risk” will over- or under-act.

Validation is a program, not a screenshot:

• Internal random split is the weakest story if the same time period leaked into both sides.
• Temporal validation (train on earlier encounters, test on later ones) is closer to real use.
• Local validation is required when the model was trained elsewhere. Another hospital’s ranking performance is not your ED.
• External or multi-site validation tests whether the relationship travels.
• Silent or prospective trial watches the score in your workflow without letting it change care, then compares predicted versus observed and measures burden.
• Decision impact asks whether the prescriptive step changed a process or outcome you care about—not only whether the AUC looked high in a slide deck.

Do not invent or demand a single unpublished cutoff (“AUC must be 0.8”). Demand a validation plan that matches intended use. A clinic no-show model that only ranks well on weekday English-preferring visits is not validated for evening interpreter-dependent sessions.

A high AUC does not equal safe deployment. Carry that sentence into every vendor demo. Ranking on last year’s inpatients does not prove:

• the score is calibrated tonight,
• false-positive pages are survivable,
• underrepresented groups are not systematically scored “low risk,”
• labels were free of leakage (for example, using a discharge disposition that is only known after the event),
• the model still works after a new EHR version or a coding initiative,
• anyone knows how to override it,
• the organization has liability, privacy, and monitoring owners.

Those are governance questions, not decoration.

Bias, leakage, and drift — why AI is not magic

Models learn the history you feed them, including injustice and documentation habits.

Training-data bias. If evening clinics, non-English preference, or Medicaid patients were rare in the training file, the model may score those encounters as “low risk” because it never saw their no-shows or deteriorations labeled well. Overbooking those sessions is an equity and operations failure wearing a math costume.

Label bias. If “deterioration” was defined as a rapid-response call, units that under-call will look low risk. If “readmission” ignores other hospitals, you will miss the bounce-back. If nurses document social needs unevenly, the model will learn documentation, not need.

Leakage. Features that are only known after the outcome (death flag, discharge status, a late ICD code) make last year’s test look brilliant and tomorrow’s use collapse. Informatics should be able to name when each feature is available in the real workflow.

Feedback loops. If high-risk patients get extra utilization, and utilization is a feature, the score can chase its own tail. If staff stop examining low-score patients, missed events never enter the label, and the model “proves” it was right.

Drift. New order sets, a pandemic, a different hospital in the system, or a documentation blitz can change the relationship between features and events. A model that is not monitored is a forgotten protocol.

Generative AI is not a validated risk model. A chatbot that summarizes a chart or drafts a note is a different tool class. It can hallucinate, is rarely calibrated as a probability, and does not become a sepsis score because someone typed “assess risk.” Do not let a demo of fluent language substitute for the validation list above. Do not treat “the model said so” as a clinician order.

Governance before the page fires

CPHIMS-level model governance is the same professional habit you use for CDS and for certified measures:

1. Intended use. Who sees the score, at what moment, to decide what? A care-management queue is not an automatic rapid-response order.
2. Owner. Clinical owner for the action, analytics owner for the math, informatics owner for the workflow build.
3. Human review. Especially for high-stakes or irreversible actions. Silent automation of medications, transfers, or restraint-like restrictions is the wrong default.
4. Threshold and capacity. Choose an operating point you can staff. Publish expected flag volume.
5. Fairness review. Stratify performance and outcomes by relevant groups you can measure honestly. Absence of a disparity analysis is not proof of fairness.
6. Privacy and minimum necessary. Scores and features are still PHI. Vendor training on identifiable production data needs a business-associate and a purpose.
7. Monitoring. Calibration, volume, override reasons, outcome among flagged and unflagged, and drift triggers that take the model offline.
8. Change control. A new version is a clinical change, not a color update on a tile.
9. Sunset. If nobody uses it, or if it harms, turn it off.

If a vendor boasts “industry-leading accuracy,” asks to auto-order rapid response, and has no local temporal validation, no calibration plot, no bias review, and no monitoring plan, the CPHIMS answer is not yet. A high AUC on last year’s file is a ranking souvenir. It is not a safety outcome.

Link the model back to sections 8.1 and 8.2: show scores in a table or calibration view that humans can audit; do not hide them in a black-green gauge. Pair deployment with process, outcome, and balancing measures. A model that raises early warning calls (process) while missing arrests stay flat and nurse overtime explodes has not automatically succeeded.

Scenarios and exam traps

Scenario. Leadership wants a weekly readmission rate, a drill-down into why heart-failure discharges bounce back, a score for next-week risk, and a staffing rule that adds a transition nurse when risk is high. Map them: descriptive, diagnostic, predictive, prescriptive. Do not call all four “the AI.”

Scenario. A vendor boasts a high AUC for a deterioration model on last year’s inpatients and wants auto-order rapid response this weekend. Require local, temporally valid testing, calibration, bias review, a human-reviewed workflow, and monitoring. Ranking on a past sample is not safe deployment.

Scenario. A no-show model trained mostly on weekday English-preferring clinic users scores evening interpreter-dependent clinics as low risk. Those sessions are overbooked and collapse. That is training-data and label bias, not proof the community “just shows up.” Govern intended use and equity before expanding overbook rules. Do not treat the score as magic.

Scenario. A model uses discharge disposition and a death flag among its “predictors” of inpatient deterioration. Those labels are not available at the time of prediction. That is leakage. Last year’s test will look brilliant and tonight’s use will fail.

Scenario. A team replaces a validated early-warning score with an untested chatbot prompt because the language “sounds clinical.” Generative fluency is not calibration, not a probability, and not an order.

Watch these traps:

1. Calling every dashboard predictive or every model AI.
2. Treating a vendor AUC as a safety outcome.
3. Skipping local temporal validation and calibration.
4. Silent automation without capacity or override.
5. Training-data bias briefed as patient behavior.
6. Leakage features that exist only after the event.
7. No drift monitoring after EHR or coding changes.
8. Inventing unpublished performance numbers to close an argument.

/practice/cphimsPractice questions with detailed explanations