9.3 Few-Shot Prompting & In-Context Learning

Few-Shot Prompting & In-Context Learning

Exam Blueprint Focus: The CCDV-F exam rigorously tests your understanding of in-context learning mechanics. You must know when to apply few-shot prompting versus zero-shot or fine-tuning, how to structure demonstrations using canonical XML schemas, how to curate diverse edge-case suites, how to build contrastive negative examples, how to detect and eliminate statistical demonstration biases (label skew, recency bias), and how to leverage prompt caching to eliminate token cost overhead.

Mechanics of Few-Shot Learning in Claude

Few-shot prompting is an in-context learning (ICL) technique wherein developers provide two or more input-output demonstration pairs within the prompt context before presenting the actual query. Unlike model fine-tuning, which alters the underlying neural network weights via backpropagation, few-shot prompting operates purely within the forward pass of inference:

• Attention Key-Value Activation Anchoring: The demonstrations populate the transformer's attention cache with concrete key-value token representations. When Claude processes the target query, its self-attention heads attend directly to the demonstration pairs, conditioning the output token probability distribution toward the demonstrated style, syntax, and reasoning patterns.
• Implicit Rule Induction: While written instructions describe rules abstractly ("be concise, extract named entities, format as JSON"), demonstrations provide empirical ground truth. Claude reconciles instructions against demonstrations, using the examples to resolve semantic ambiguities.

Zero-Shot vs. Few-Shot vs. Fine-Tuning: Architectural Trade-Offs

Choosing the appropriate learning paradigm is a critical architectural decision evaluated on the CCDV-F exam.

---

Structuring Few-Shot Demonstrations with XML Tags

A frequent error in naive prompt engineering is formatting examples as unstructured plaintext:

Input: The patient presents with acute rhinitis.
Output: Diagnosis: Common Cold | Severity: Mild

Input: Severe lumbar disc herniation with radiculopathy.
Output: Diagnosis: Herniated Disc | Severity: High

Input: Bilateral conjunctivitis.

This unstructured approach triggers the continuation trap: Claude frequently treats the prompt as an incomplete document and continues generating hypothetical additional input-output pairs (Output: Diagnosis: Eye Infection... Input: Migraine with aura...) rather than resolving the user's active query.

The Canonical Anthropic Few-Shot XML Schema

To prevent continuation loops and unambiguously isolate reference demonstrations from runtime execution, Anthropic recommends encapsulating all examples within structured XML tags:

<examples>
  <example id="1">
    <input>
      {"event": "AUTH_FAIL", "ip": "192.168.1.50", "attempts": 12, "user": "root"}
    </input>
    <ideal_output>
      <threat_assessment>
        <severity>CRITICAL</severity>
        <category>BRUTE_FORCE</category>
        <action>BLOCK_IP_IMMEDIATE</action>
        <confidence_score>0.98</confidence_score>
      </threat_assessment>
    </ideal_output>
  </example>
  <example id="2">
    <input>
      {"event": "AUTH_FAIL", "ip": "10.0.0.12", "attempts": 1, "user": "jsmith"}
    </input>
    <ideal_output>
      <threat_assessment>
        <severity>LOW</severity>
        <category>ANOMALOUS_LOGIN</category>
        <action>LOG_AUDIT_TRAIL</action>
        <confidence_score>0.25</confidence_score>
      </threat_assessment>
    </ideal_output>
  </example>
</examples>

By wrapping demonstrations in <examples> and <example>, you explicitly define the boundaries of the reference set. Claude recognizes that the examples represent static reference contracts, while the subsequent <query> represents the live operational turn.

---

Selecting Optimal Examples: Diversity, Edge Cases & Refusals

In few-shot engineering, quality and variance vastly outperform quantity. Providing 3 to 5 carefully selected, structurally diverse demonstrations consistently yields higher accuracy than providing 20 redundant examples that repeat the same syntax.

The Three Pillars of Few-Shot Curation

1. Input Structural Diversity: If your few-shot suite only includes short, 10-word sentences, Claude will struggle when presented with a 200-word paragraph containing nested clauses. Include examples across varying lengths, vocabulary registers, and syntactic complexities.
2. Boundary & Edge-Case Coverage: Standard queries rarely break generative pipelines; edge cases do. Include demonstrations that illustrate:
   • Missing or null data fields: Showing how the model should output "value": null or <status>DATA_UNAVAILABLE</status> rather than hallucinating plausible values.
   • Malformed inputs: Demonstrating graceful handling of misspelled headers or truncated records.
   • Ambiguous queries: Showing how Claude should output a clarifying request or a calibrated confidence score.
3. Explicit Refusal & Safety Boundaries: At least one demonstration should depict a query that falls outside the system's operational boundaries or safety guidelines. Demonstrating how to refuse a request—using polite, objective, non-preachy language—prevents Claude from generating lecturing or unhelpful refusal boilerplate at runtime.

---

Negative Examples & Contrastive Demonstrations

While standard few-shot demonstrations show Claude what to do, complex enterprise domains often require steering Claude away from subtle anti-patterns (such as over-verbose justifications, speculative extrapolation, or legal liability traps). Contrastive few-shot learning provides paired examples of what not to do alongside the corrected version.

<examples>
  <example type="contrastive">
    <user_query>
      Analyze whether our API gateway should allow CORS requests from 'https://*.partner-domain.com'.
    </user_query>
    <bad_output>
      Yes, you can configure CORS with a wildcard subdomain. It's convenient for partner integrations
      and will allow all partner services to connect without individual domain whitelisting.
    </bad_output>
    <rationale>
      The bad output ignores critical security vulnerabilities. Wildcard subdomains in CORS headers allow
      any compromised or maliciously registered subdomain under that parent domain to bypass the Same-Origin Policy.
    </rationale>
    <good_output>
      REJECT CONFIGURATION. Wildcard subdomain matching in Access-Control-Allow-Origin headers is a security anti-pattern.
      Remediation: Implement an explicit server-side origin whitelist validation function that matches fully qualified
      domain names (FQDNs) exactly before returning the validated origin in the response header.
    </good_output>
  </example>
</examples>

Why Contrastive Demonstrations Work

Negative examples combined with an explicit <rationale> block activate Claude's comparative reasoning capabilities. The model internalizes the specific decision boundary that separates the flawed response from the acceptable response, providing far sharper steering than positive examples alone.

---

Mitigating In-Context Biases in Few-Shot Prompts

Autoregressive models are extraordinarily sensitive to the statistical properties of their context windows. Poorly designed few-shot suites introduce insidious behavioral biases that degrade production reliability:

1. Label Skew (Majority Class Bias)

If a 5-shot sentiment classifier contains 4 positive examples and 1 negative example, Claude's attention mechanism absorbs an artificial prior probability distribution heavily skewed toward "POSITIVE." In production, the model will misclassify neutral and mildly negative inputs as positive.

• Mitigation: Enforce strict mathematical balance across all target classification categories (e.g., exactly 2 Positive, 2 Neutral, 2 Negative).

2. Recency Bias

Claude's self-attention heads assign slightly higher attention weight to tokens closest to the active query turn. If the final demonstration in your <examples> block is "URGENT", the model exhibits an empirical bias toward classifying borderline live queries as "URGENT."

• Mitigation: Avoid placing edge cases or high-severity classes exclusively at the terminal end of the example suite. For critical pipelines, shuffle demonstration order dynamically at runtime.

3. Length & Formatting Overfitting

If all demonstrations in your few-shot prompt contain exactly three bullet points of 12 words each, Claude will treat this structural artifact as a mandatory contract. It will compress or artificially pad all runtime responses into three 12-word bullets, regardless of query complexity.

• Mitigation: Deliberately vary response lengths, sentence counts, and internal structures across demonstrations.

---

Token Overhead, Latency & Prompt Caching Integration

The primary drawback of few-shot prompting is token overhead. A robust suite of 8 detailed demonstrations can easily consume 2,500 to 4,000 input tokens. In high-throughput production systems, this overhead increases both per-request billing and time-to-first-token (TTFT) latency.

The Prompt Caching Solution

Anthropic Prompt Caching completely transforms the economics of few-shot prompting. By placing static few-shot demonstrations inside the cached prompt prefix, developers achieve the accuracy benefits of large demonstration suites with virtually zero ongoing latency or cost penalty.

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are an enterprise code reviewer. Enforce the exact standards demonstrated below:\n\n<examples>\n... [3,200 tokens of diverse, curated code review examples] ...\n</examples>",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "Review this pull request: [diff text]"
    }
  ]
}

Economic and Performance Impact

• Cost: The initial request writes the 3,200-token demonstration suite to the cache (priced at a 25% premium over standard input). All subsequent requests read from the cache at a 90% discount relative to standard input token prices.
• Latency: Because the key-value activations for the 3,200 demonstration tokens are already computed and held in GPU memory, Claude skips forward-pass computation for the examples, reducing TTFT by up to 80%.

---

Exam Watchouts & Common Pitfalls

1. The Continuation Trap: Providing examples without XML boundaries, causing Claude to generate new imaginary examples instead of answering the query.
2. Demonstration Contradictions: Providing few-shot examples whose formatting contradicts instructions in the system prompt. When instructions and demonstrations conflict, Claude frequently follows the demonstrations, leading to unexpected schema violations.
3. Uncached Few-Shot Bloat: Deploying 10+ few-shot examples in interactive microservices without prompt caching, causing high latency and inflated operational costs.
4. Overfitting to Example Specifics: Using examples with identical entity names or values (e.g. all examples using "ACME Corp"), which can cause Claude to hallucinate "ACME Corp" in unrelated production answers.