15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies

Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies

Quick Answer: In Azure AI Search, a vector query returns k nearest neighbors even when they are weakly related. Use a vector similarity threshold (preview) when you must drop junk before fusion. Chunk around 512 tokens with ~25 percent overlap, or start the Text Split skill at 2,000 characters with 500-character overlap. Prefer parent-child or semantic chunking so the generator sees a coherent passage, not a truncated sentence or an entire manual.

Domain 5 of Exam AI-300 — Optimize generative AI systems and model performance (10–15%) — opens with optimize retrieval performance: similarity thresholds, chunk sizes, and retrieval strategies. A retrieval-augmented generation (RAG) pipeline is only as good as the passages you hand the model. If the index returns the wrong chunk, groundedness and relevance evaluators in Microsoft Foundry will fail even when the chat model is excellent.

Why retrieval is the first knob

A Foundry agent or prompt-flow RAG node typically does four things: embed the user query with the same embedding model used at index time, search Azure AI Search (vector, keyword, or hybrid), pack the top passages into the prompt, and call a chat completion or Responses API model. Hours of prompt rewriting cannot fix a corpus that was sliced into 80-character fragments or into 40-page blobs that exceed text-embedding-3-small’s 8,192-token input limit and get truncated.

Treat retrieval as an MLOps asset. Store chunking parameters (size, overlap, splitter, parent key) next to the indexer skillset in Git. Rebuild the index when those parameters change. Do not “tune RAG” by only raising temperature.

Chunk size and overlap

Chunking partitions source documents so each embedding represents one semantically useful unit and stays under model token limits. Microsoft’s Azure AI Search chunking guidance starts with a fixed-size chunk of 512 tokens (about 2,000 characters for common English tokenizers) and an overlap of 25 percent (128 tokens). Overlap copies the tail of one chunk onto the head of the next so a sentence that straddles a boundary is not lost.

The built-in Text Split skill (used by integrated vectorization) splits on pages (multi-sentence windows) or sentences. For pages, maximumPageLength is the target character or token budget and pageOverlapLength must be less than half of that maximum. Microsoft’s common starting values when measuring characters are maximumPageLength 2000 and pageOverlapLength 500. Sentence mode produces many more, smaller chunks (a NASA e-book sample jumped from tens of page-chunks to more than 13,000 sentence chunks).

Too-small chunks match on isolated nouns but omit the sentence that defined them (“the waiting period is 90 days” without saying which benefit). Too-large chunks average many topics into one vector, so cosine similarity becomes mushy, indexing cost rises, and you may hit the embedding 8,192-token cap and silently drop the tail. Chat models also have context budgets: stuffing eight 4,000-token parents into a prompt crowds out the question.

Practical overlap rules:

• Start near 10–25 percent. Microsoft’s 25 percent / 128-token figure is a documented starting point, not a law.
• Highly structured tables and SKU lists often need less overlap; narrative policies need more.
• If overlap is larger than the next chunk, consecutive chunks can be identical and you waste index quota.
• Measure token length with the same tokenizer the embedding model uses (for Azure OpenAI embeddings, cl100k_base is the usual estimate), not raw character count alone.

top-k versus a similarity threshold

The query-time parameter k is how many nearest neighbors the vector engine must return. Hierarchical Navigable Small World (HNSW) approximate search and exhaustive k-nearest neighbors (eKNN) both honor k. Microsoft is explicit: if at least k documents exist, the engine always returns k results, even for a nonsense query. Weak neighbors simply have low @search.score (cosine similarity for Azure OpenAI embeddings typically sits between about 0.333 and 1.00).

That “always return k” behavior is useful for recall (you would rather over-retrieve and let a ranker drop junk) and dangerous for RAG (the model will happily quote an unrelated policy). Two complementary controls:

1. Keep k, then trim in the app. Retrieve k = 20–50, drop chunks below a cosine cutoff in your code, then send the survivors to the LLM. Simple, but you pay for the extra search work.
2. Vector similarity threshold (preview). On recent preview Search APIs, each vectorQueries item can include "threshold": { "kind": "vectorSimilarity", "value": 0.8 }. Matches below that similarity are excluded before Reciprocal Rank Fusion, and the result count may fall below k. Use this when junk neighbors pollute hybrid fusion.

Do not apply a 0.8 cosine cutoff to a hybrid @search.score. After Reciprocal Rank Fusion (RRF), scores often sit near 0.03 even for strong matches. Thresholds belong on the vector subquery’s native similarity, not on the fused RRF number. Microsoft also warns that @search.rerankerScore distributions from semantic ranker shift slightly across infrastructure updates, so do not encode an ultra-granular reranker cutoff.

For RAG, a typical production pattern is: vector or hybrid k = 50 when semantic ranker will run (the ranker consumes up to 50 inputs), then top = 5–10 passages in the prompt. Semantic ranker starved of fewer than 50 candidates cannot recover recall you never retrieved.

Parent-child and semantic chunking

Semantic chunking follows document structure: H1/H2 headings, paragraphs, and Markdown sections rather than a blind character window. Azure AI Search can drive this with the Document Layout skill (structure from Document Intelligence-style layout) plus Text Split, or with the Azure Content Understanding skill that emits Markdown and semantically coherent fragments. The payoff is a chunk that still “means” one procedure, one table, or one FAQ answer.

Parent-child (sometimes called small-to-big) indexing stores:

• Child documents: the embeddable chunk, plus parentId, heading path, page number, and source URI.
• Parent documents: the full section or file, not necessarily embedded, or embedded separately for a second-stage fetch.

At query time you retrieve children (precise match), then expand to the parent or to ±1 neighboring chunks so the generator sees defined terms and exceptions. One-to-many blob indexing (JSON arrays or Markdown sections as separate search documents) is the indexer-native way to materialize children.

Exam scenario: A benefits RAG bot is citing the wrong waiting period. Logs show k = 8 cosine neighbors, all from a 60-page PDF stored as one vector. The embedding mixed “dental waiting period 90 days” with “medical waiting period 30 days.” Re-chunk with Text Split pages at 2,000 characters / 500 overlap, store heading and parentId, retrieve children, and expand to the parent H2 section before generation. Do not first swap the GPT deployment.

Common trap: Raising k from 5 to 80 “to be safe” without a threshold, then pasting all 80 chunks into the prompt. You blow the context window, raise token cost, and the model attends to distractors. Pair a moderate k with overlap, parent expansion of a few winners, and (if needed) a vector similarity threshold before fusion.

Retrieval strategy checklist

Choose a strategy, then change one parameter per experiment:

• Vector-only: Conceptual questions (“how do we handle a lapse?”). Always-k means you still need a threshold or a groundedness check.
• Keyword / BM25-only: SKUs, form numbers, error codes, people’s names, dates.
• Hybrid (BM25 + vector, fused with RRF): Default for mixed enterprise corpora (section 15.3).
• Hybrid + semantic ranker: Microsoft’s published benchmarks often put this combination first; it is a paid rerank of the top 50, not a second index.
• Filtered retrieval: filter plus vectorFilterMode preFilter (default, shrinks the HNSW surface; use for security trimming) or postFilter (trim after search; can starve semantic ranker).
• Parent expansion: Retrieve children, hydrate parents, then generate.

Tune in this order: chunk graph → embedding model (section 15.2) → hybrid/semantic (section 15.3) → k and threshold → prompt. Measure with recall@k and the Foundry document retrieval evaluator (section 15.4) while holding the generator constant.