2.4 AI Hallucinations, Limitations & Failure Modes

2.4 Hallucinations, Limitations, and Failure Modes

Core principle: A language model can produce a coherent answer without evidence that the answer is true. Treat important output as a claim to verify, not as an authority to accept or reject solely because of tone.

---

What Hallucination Means

A hallucination is generated material that is false, unsupported by the cited or supplied evidence, internally inconsistent, or presented with unjustified certainty. An autoregressive language model is trained to predict plausible token sequences. It does not automatically execute a fact-check, observe the product, or know which local policy is current.

The mechanism is not the only cause. Errors can enter through stale training data, missing context, ambiguous instructions, bad retrieval, an incorrect tool result, a poisoned source, or an invalid assumption supplied by the user. A RAG answer may quote the wrong passage; an agent may faithfully use a broken API; a summarizer may omit a dissenting view.

Common Failure Modes

A malicious party may register a commonly invented package name, creating a supply-chain path sometimes called slopsquatting. Never install a generated dependency merely because its name sounds plausible.

Controls and Their Limits

Better prompts can request sources, assumptions, alternatives, and explicit unknowns. Lower temperature can reduce variation. RAG can supply current evidence. Self-critique can expose an inconsistency. Each is useful, and none guarantees truth.

Use independent, task-specific checks:

• factual claims: current primary source and claim-level citation;
• arithmetic or counts: calculator, query, or reproducible script;
• code: compiler, tests, static analysis, dependency and security review;
• product behavior: observed acceptance and operational tests;
• consequential people decisions: authorized policy, appropriate expertise, documented oversight, and appeal or correction paths.

A model should be allowed to abstain when evidence is missing. Ask it to label what is known from supplied sources, what is an inference, and what requires confirmation. Confidence expressed in prose is not calibrated probability.

Risk-Based Verification

Verification effort should reflect consequence, reversibility, exposure, and uncertainty. A brainstormed Retrospective theme may need a facilitator's read. A generated authentication change needs code review, tests, security checks, and operational safeguards. A model affecting health, credit, employment, or safety may require specialized legal, ethical, and domain review beyond this study guide.

Automation can be appropriate when inputs and outputs are bounded and failures are detectable and recoverable. Scrum does not impose a universal manual review step on every tool operation. It does require that the people holding accountabilities do not pretend a tool owns the result.

Neutral Questions Reduce Leading-Premise Risk

Instead of “explain why we should skip Retrospectives,” ask: “Using the Scrum Guide, explain the purpose of the Sprint Retrospective, the risks of not holding it, and options for keeping it productive within its timebox.” Neutral framing reduces pressure to confirm a premise, but the answer still needs comparison with the source.

Correct Scrum Accountability

Developers are accountable for creating a usable Increment and instilling quality by adhering to the Definition of Done. The Product Owner is accountable for maximizing value and effective Product Backlog management; the Product Owner does not universally accept each story or approve every release. The Scrum Master is accountable for establishing Scrum and team effectiveness. The whole Scrum Team is accountable for creating a valuable, useful Increment every Sprint.

If generated code contributes to a defect, inspect the system and improve it rather than blaming a tool as though it were a team member. Update tests, the Definition of Done, permissions, sources, or workflow based on evidence. Separate Scrum accountabilities from any legal or contractual liability analysis.