22.1 CAC, NLP and AI: Context, Oversight and Performance
Key Takeaways
An extracted concept, software suggestion, provider diagnosis and reportable code are separate stages requiring the applicable professional review.
Preserve negation, uncertainty, time and the person experiencing the condition; historical or family mentions alone do not establish a new acute diagnosis.
Confabulation, automation bias and performance disparities are different risks. Verify actual evidence and avoid assuming that a vendor score or approval click resolves them.
The 2026 query brief allows pre-delivery review or ongoing structured auditing; automated delivery retains organizational accountability and requires regular structured auditing.
Precision is TP/(TP+FP), recall is TP/(TP+FN), and false-positive rate is FP/(FP+TN). An audit of flags alone cannot determine recall.
CAC, NLP and AI: Context, Oversight and Performance
Computer-assisted coding (CAC) can propose codes or identify text for review. Natural language processing (NLP) can identify concepts and their context in narrative records. Generative AI can draft summaries or query language. These functions overlap in some products, but they are not a mandatory sequence of technological “generations.” Evaluate the actual intended use, inputs, output and workflow of the local tool.
An extracted phrase, a proposed diagnosis and a reportable code are different things. Finding the word “stroke” in a chart does not establish that the patient has a current acute infarction. The responsible professionals still must apply the clinical, documentation, query and coding requirements appropriate to the task.
Read beyond the extracted concept
Clinical NLP must preserve context. The original ConText research illustrates why negation, time and the person experiencing a condition matter; it does not establish the performance of every contemporary product. ConText primary study, introduction and algorithm description
| Illustrative source wording | Information a reviewer must preserve |
|---|---|
| “Denies chest pain” | The symptom is negated in this statement |
| “Father had a stroke” | The condition refers to a family member |
| “Stroke in 2019, no residual deficits” | A historical event is not automatically a new acute event |
| “Return if fever develops” | A conditional future event is not a current finding |
| “Pneumonia considered; further evaluation pending” | The statement expresses uncertainty at this point in the course |
These examples are reading exercises, not automatic coding instructions. A clinical report may use “history” to describe a condition that remains relevant, and reportability depends on the complete encounter and applicable rules. Similarly, inpatient uncertain-diagnosis rules apply to specified discharge documentation and settings; an early differential or a software label cannot be substituted for those requirements. FY2027 ICD-10-CM guidelines, Sections III.C and IV
Worked review: An automated summary converts “mother had diabetes” into “patient has diabetes.” Locate the original sentence, verify the person and encounter, correct the output through the authorized workflow and report the attribution defect for investigation. A provider-facing query for a new patient diagnosis cannot be justified by the family-history sentence alone. Review any separate current evidence before deciding whether a different clarification is warranted.
Recognize different failure mechanisms
Extraction or mapping error: The tool may lose a negative modifier, confuse the source person, attach the wrong date or map a nonspecific term to an unsupported specific concept. Check the original text rather than trusting the highlighted keyword.
Confabulation: A generated summary may introduce a finding, relationship or citation absent from its inputs. Plausible prose is not evidence. Verify clinical observations against the actual health record and verify cited guidance against its source. NIST identifies confidently erroneous content as a generative-AI risk, including in consequential healthcare uses. NIST Generative AI Profile, §2.2
Automation bias: A person may defer to the output without evaluating it. A nominal approval click provides little protection if the reviewer cannot inspect the source or lacks the time, training and authority needed to challenge the result. Conversely, rejecting a useful tool solely because it is automated also avoids evaluating its actual performance. NIST Generative AI Profile, §2.7
Directional design or performance disparities: A system can overproduce a favored diagnosis or perform differently across note types, services, languages or populations. Investigate the actual outputs and configuration. Do not assume every vendor intentionally tunes software to upcode or that one aggregate score represents every group. NIST distinguishes harmful bias and performance disparities from other AI risks. NIST Generative AI Profile, §2.6
An abnormal laboratory value or medication can be a review trigger without establishing a diagnosis. For example, albumin alone is not an accepted nutrition-status marker for diagnosing malnutrition. A high provider agreement rate with albumin-triggered queries does not validate that premise. Check the clinical evidence and query construction; agreement, clinical support and code accuracy are separate outcomes. ASPEN position paper on visceral proteins
Apply query standards to the function
The 2026 AHIMA/ACDIS brief covers technology-generated prompts, nudges and advisories when they function as patient-specific documentation queries. A vendor label does not exempt the communication. It distinguishes passive reference information from a communication directing consideration of a specific patient diagnosis or documentation option.
The brief calls for organizational vetting, query and compliance involvement, training and ongoing monitoring. Human oversight may occur before delivery or through ongoing structured auditing. Automated delivery without pre-delivery review is therefore not categorically prohibited by the brief. It requires structured regular auditing and continued organizational accountability. Local policy may choose a stricter review process based on risk.
Evaluate relevant, traceable indicators, neutral wording and supported response options, including a way to give an alternate explanation. Test the provider-visible title and interface as well as the body. Review selection, drafting and delivery controls together; a compliant template can still be populated with the wrong patient's data. 2026 compliant-query guidance, §XI
Measure a defined task
First define the unit and reference standard. A “positive” could mean that a particular record contains a legitimate opportunity for review under specified criteria. It does not mean that a diagnosis has been established or that a provider must answer yes. Use qualified independent review and resolve disagreements in the reference labels.
For an illustrative audit of 100 records, suppose reviewers classify 30 as containing the defined opportunity. The tool flags 40 records, of which 24 are true opportunities:
| Reference review | Tool flags | Tool does not flag | Total |
|---|---|---|---|
| Opportunity present | 24 true positives | 6 false negatives | 30 |
| Opportunity absent | 16 false positives | 54 true negatives | 70 |
| Total | 40 | 60 | 100 |
- Precision: true positives / all positive flags = 24 / (24 + 16) = 60%. How many flagged records actually have the defined opportunity?
- Recall or sensitivity: true positives / all actual positives = 24 / (24 + 6) = 80%. How many actual opportunities did the tool find?
- False-positive rate: false positives / all actual negatives = 16 / (16 + 54), approximately 22.9%. How often did it flag records without the opportunity?
The 40% of flags that are false positives is not the false-positive rate; it uses a different denominator. An audit limited to flagged records can estimate precision but cannot determine recall without examining a suitable set of unflagged records. A provider agreement rate answers neither question.
These numbers describe this illustrative task and sample. They do not prove that 60% precision is acceptable, establish a contractual target or measure every dimension of query compliance. Assess both missed problems and unnecessary prompts, account for the sampling design and uncertainty, and compare relevant subgroups. NIST's framework is voluntary risk-management guidance, not a federal requirement for an 85% CDI precision threshold or a universal quarterly audit schedule. NIST AI Risk Management Framework
Monitor the deployed workflow
Assign owners for clinical content, coding rules, query compliance, technical operation and privacy/security. Establish the intended setting, permitted data, version, evaluation method and response to defects. Confirm that authorized users can inspect meaningful source evidence and report problems. Use approved systems for PHI; do not paste records into an unapproved consumer AI service merely to obtain a second opinion.
Reassess performance after relevant changes such as a model update, new note template, data-interface change or coding-year transition. A local decline may reflect changed inputs, reference labels or workflow as well as model behavior. Investigate before changing thresholds. Keep records sufficient for the organization's audit, retention and incident-response obligations; “permanent logging of every suggestion” is not a universal retention rule.
If an output is defective, address the immediate communication and determine the affected scope with authorized owners. Some problems require a template fix; others require a system pause, broader case review or incident response. Do not assume that a signature or vendor assurance makes the issue harmless. Equally, an inaccurate suggestion alone does not establish all FCA elements or automatically trigger a per-claim penalty; use the conduct and legal analysis taught in Section 21.1. 2026 query guidance, §XI; NIST Generative AI Profile, §§2.4 and 3
An NLP summary changes “mother had diabetes” into “patient has diabetes.” What is the best immediate response?
Verify the source and person, correct the output through the authorized workflow and report the attribution defect; review separate current evidence before considering any patient-specific clarification.
Send a diabetes query based on the summary alone because a family-history sentence is sufficient evidence of the patient’s diagnosis.
Exclude every family-history section from all future review, without evaluating the tool’s intended task or the record.
Assign the patient’s code after a second AI system repeats the same summary, without checking the original note.
A software proposal delivers patient-specific diagnostic queries without a separate human approval before each delivery. Under the 2026 AHIMA/ACDIS brief, which response is correct?
The brief categorically prohibits the design, regardless of any structured auditing, governance or local risk assessment.
Vendor certification removes the organization’s responsibility for the content and consequences of delivered queries.
Evaluate compliant content and organizational controls; the brief permits oversight before delivery or through ongoing structured auditing, with regular structured auditing for automated delivery without pre-delivery review.
Review is needed only if the provider rejects the suggested diagnosis or the query decreases reimbursement.
Independent review of 100 records finds 30 with a defined documentation opportunity. A tool flags 40 records, including 24 true positives and 16 false positives; it misses 6 opportunities and correctly leaves 54 negative records unflagged. Which result is correct?
Precision is 80%, recall is 60%, and false-positive rate is 40%.
Precision is 60%, recall is 80%, and false-positive rate is approximately 22.9%.
Precision is 24%, recall is 40%, and false-positive rate is 16%.
Precision is 60%, recall is 80%, and false-positive rate is 40%.
Sections you finish are checked off in the contents.