18.2 Internal CDI Quality Audits and Reviewer Calibration

Internal CDI Quality Audits and Reviewer Calibration

An internal audit asks whether the documentation process worked as intended and what should change when it did not. Review volume, query response and agreement percentages cannot answer that question alone. Examine the records, the actual queries, provider responses and coding decisions using the rules applicable to the encounter.

The 2026 AHIMA/ACDIS query brief, Section XIII, calls for organizational policies addressing audit methodology, frequency, sample selection and evaluation criteria. It does not prescribe a national 100-point CDI scorecard, 10–15 records per specialist each month or a universal automatic-zero scoring rule. The OIG General Compliance Program Guidance, Element 6, supports risk-informed audits and monitoring; that guidance is voluntary and does not guarantee immunity from enforcement or payer review.

Define the question and the evidence

The following are useful audit perspectives, rather than four legally required categories:

Preserve the encounter and applicable rule version. A discrepancy may reflect a true coding error, an unsupported query, new results after the concurrent review, an incomplete handoff or a reasonable issue needing further clinical clarification. Record which explanation the evidence supports rather than assigning every difference to the specialist or coder.

Evaluate substantive query integrity

Check that the query faithfully represents the available record, including relevant evidence that may oppose the suggested interpretation. Clinical indicators should retain their source and context. A multiple-choice query needs supported options and an open alternative; the 2026 brief does not require several competing diagnoses or a fixed “Other/Unable/Ruled out” trio in every case.

Leading language or pressure based on reimbursement or quality outcomes is a substantive concern requiring correction. A facility may designate such a concern as a critical failure in its own rubric. That local scoring decision is different from claiming that AHIMA mandates a zero score for the entire record. The brief also rejects treating isolated technical deviations as a rigid stand-alone denial or fraud checklist.

Example: “Please document respiratory failure to obtain an MCC” directs a preferred clinical answer for a payment purpose. The auditor should identify both defects, route remediation under policy and assess any affected documentation or coding. A provider signature does not make that wording acceptable. Conversely, the presence of just one clinically supported diagnostic option is not automatically leading when the overall question is neutral and includes the required open alternative.

For a missed-opportunity review, ask whether there was a supported, useful documentation question at the time of review. A dietitian's assessment may support provider clarification about malnutrition; it does not permit the auditor to establish the diagnosis independently. Likewise, meeting an AKI criterion can be relevant evidence, but does not make the auditor the diagnosing provider. Apply the FY2027 coding guidelines, including I.A.19, I.B.14 and Section III, alongside clinical review.

Match sample design to the purpose

Define the audit population, period, unit, selection method and exclusions before opening records. A unit might be a query, an encounter or a specific decision; the denominator must match. Include human and technology-generated queries when they are in scope.

For targeted learning, deliberately select records with a known concern, such as a recurring response-documentation problem. This is useful for understanding the problem, but its error proportion is not automatically an estimate of the whole department's performance. For population estimation, use an appropriate probability-based design with a defined sampling frame and statistical expertise when needed. Sample size depends on the question, variability, precision and intended inference, not a universal CDI chart count.

If using strata, define them so records are not unintentionally counted twice. “Queried,” “non-queried” and “high risk” can overlap: a high-risk record may also be queried. Separate those concepts or explicitly manage the overlap. An unweighted result from deliberately oversampled high-risk cases should not be presented as the overall department rate.

Hypothetical example: An auditor selects ten records specifically because a tracking report flagged possible leading wording. Four have a confirmed substantive defect. Four of ten is 40% of this selected sample. It does not establish that 40% of all departmental queries are defective. The next action may be to correct those queries and assess whether a broader, appropriately designed review is needed.

Measure reviewer agreement without confusing it with accuracy

Calibration begins with a shared decision definition and the same information available to each reviewer. Have reviewers rate the cases independently before discussing disagreements. Then reconcile the reasons, update the decision guide when appropriate and reassess consistency on a suitable set of cases. Consensus reached after discussion is useful education, but it is not independent pre-discussion agreement.

For two reviewers classifying the same records into nominal categories, Cohen's kappa can supplement raw agreement. It compares observed agreement with an expected agreement calculated from the reviewers' marginal category proportions under the specified independence model:

$\kappa = \frac{P_o-P_e}{1-P_e}$

It does not literally eliminate all chance or prove that the shared decisions are clinically correct. Two reviewers can agree and both be wrong. Category distributions and the case mix affect interpretation. A different design, such as multiple raters or ordered severity categories, may need a different or weighted statistic rather than automatic use of ordinary two-rater kappa. See Landis and Koch's original methodological paper, Section 3.2, for the underlying agreement framework.

Worked two-reviewer example

Both reviewers independently judge whether a clarification opportunity exists in 100 hypothetical records:

Observed agreement is the diagonal: $(40+45)/100 = 0.85$. Reviewer A uses Yes and No equally; B uses Yes for 45% and No for 55%. Thus:

$P_e=(0.50\times0.45)+(0.50\times0.55)=0.50$

$\kappa=\frac{0.85-0.50}{1-0.50}=0.70$

Landis and Koch used “substantial” for 0.61–0.80, but explicitly described their divisions as arbitrary benchmarks for discussion. Those labels do not establish a mandatory CDI pass score, staffing suspension or retraining threshold. Examine the 15 disagreements, their clinical importance, the sample and the local evaluation criteria. The result describes agreement in these ratings, not validated accuracy across every service line.

Close the audit loop

Use a scorecard with defined, reproducible items, an explicit denominator and a separate description of substantive findings. A numeric average should not conceal a serious concern. Explain any critical-item rule as local policy, and distinguish not-applicable items from failures.

Give the reviewer concrete evidence, the applicable source and the required correction. Assign responsibility for record or query remediation, coding review, template changes or education as appropriate. Set follow-up timing based on risk and policy, then check whether the change worked. Neither an arbitrary quarterly score below 90% nor a single kappa label supplies a complete corrective-action plan.