5.1 Embeddings & Vector Similarity Search in Cosmos DB

Why Vector Search Matters for AI-200

Many Azure AI solutions need semantic retrieval: find documents that mean the same thing as a user question, not only documents that share exact keywords. That capability depends on vector embeddings—numeric arrays produced by an embedding model (for example Azure OpenAI text-embedding-3-small or text-embedding-3-large) that place similar meanings near each other in multi-dimensional space.

Azure Cosmos DB for NoSQL can store those embeddings next to the original chunk text, metadata, and partition key in a single JSON item. You then run vector similarity search in the same database that already powers your operational reads and writes. For the exam, treat this as the data-plane pattern for retrieval-augmented generation (RAG): embed the query, search for nearest neighbors, pass retrieved chunks to a chat or completion model, and answer with grounded context.

Storing vectors in Cosmos DB reduces synchronization work. When a product description, support article, or meeting transcript changes, one upsert updates both the human-readable fields and the embedding. You avoid a dual-write pipeline to a separate vector store—unless product requirements force that split.

---

Embedding Shape Inside a Cosmos DB Item

A typical RAG chunk item looks like this conceptually:

• Partition key such as /tenantId or /sourceId for scale-out and RU isolation
• Content fields: title, content, sourceUri, chunkIndex, timestamps
• Embedding field: contentVector — an array of floating-point numbers whose length matches the embedding model’s dimensions (for example 1536 for many OpenAI small models)

The embedding must be generated before or during write. Cosmos DB does not invent vectors for you; your application (or a change-feed worker covered in the next section) calls an embedding API and persists the result.

Consistency Rules You Must Remember

1. Dimensions are fixed per vector path. Every item that uses /contentVector must supply an array of the same length declared in the container’s vector embedding policy.
2. Distance function must match how the model was trained. Cosine similarity is common for normalized text embeddings; euclidean and dot product are also supported when you configure them explicitly.
3. Null or missing vectors are not searchable. Items without a populated vector path simply will not rank in similarity results for that path.

---

Vector Embedding Policy

When you create (or later update, where supported) a container for vector workloads, you define a vector embedding policy. That policy tells the engine which JSON paths hold vectors and how to interpret them.

You can declare multiple vector paths on one container when different properties store different embedding spaces (for example a title embedding and a body embedding). Each path needs its own dimensions and distance function settings that match how those vectors were produced.

---

Vector Index Types: Flat, Quantized Flat, and DiskANN

Cosmos DB for NoSQL offers specialized vector indexes. The index type is a deliberate trade-off among recall, latency, storage, and scale.

DiskANN keeps quantized (compressed) representations efficiently accessible while using SSD-backed structures so the index can grow far beyond what an in-memory-only index would allow. It supports ongoing inserts, updates, and deletes without forcing a full rebuild for every change—critical for operational AI apps that continuously ingest content.

For AI-200 scenario questions that mention millions of chunks, low latency, and cost-effective approximate search, prefer DiskANN. Reserve flat for demos, tiny knowledge bases, or cases that explicitly require exact search.

Indexing Policy Hygiene

Exclude the vector property from the default (non-vector) index. Example concept: add /contentVector/? to excluded paths in the indexing policy. Vectors already participate in the specialized vector index; also indexing them as ordinary arrays wastes storage and request units (RUs) without helping VectorDistance queries.

---

Executing Similarity Search for Semantic Retrieval

At query time:

1. Call your embedding model with the user question (and optional conversation context).
2. Pass the resulting float array as a parameterized value @queryVector.
3. Run a NoSQL query that ranks by distance and limits results.

A representative pattern:

SELECT TOP 5 c.title, c.content, c.sourceUri
FROM c
WHERE c.tenantId = @tenantId
ORDER BY VectorDistance(c.contentVector, @queryVector)

Key exam ideas:

• VectorDistance returns a distance score according to the container’s configured distance function; lower distance means closer (more similar) for typical cosine/euclidean setups used in retrieval.
• TOP N (or an equivalent limit) is how you implement “retrieve the top-k chunks” for RAG.
• Filters and vector ranking compose. You can combine equality or range filters (tenant, language, product line, ACL tags) with vector ordering so retrieval stays tenant-safe and relevant.
• Cross-partition queries may be required when the semantic corpus spans many partition key values; design partition keys so hot tenants or sources stay balanced.

RAG Pipeline Role

In a full Azure AI solution, Cosmos DB similarity search is usually the retriever step:

1. User asks a question in your app or Azure AI Foundry / OpenAI chat flow.
2. App embeds the question.
3. Cosmos DB returns top-k chunks via DiskANN-backed VectorDistance.
4. App builds a grounded prompt (system instructions + retrieved chunks + question).
5. Chat model generates the answer; optionally log citations from sourceUri.

Because vectors live with documents, hybrid patterns—keyword filters, metadata boosts, or later hybrid full-text + vector ranking where available—can sit in the same query surface without shipping raw text to a second store solely for search.

---

Operational Design Choices

When to embed online vs offline. Batch jobs can embed historical corpora. Online paths embed on ingest. Many production designs use the change feed (next section) so new or updated items automatically receive embeddings and become searchable without blocking the write API on the embedding call.

Dimension and model upgrades. Changing embedding models often changes dimensions. That typically means a new vector path or a carefully planned re-embed and re-index of the corpus—not a silent in-place resize of an existing policy path.

RU and latency. Vector search costs RUs like other queries. DiskANN and sensible TOP N keep retrieval affordable. Oversized TOP N, missing filters, and poorly chosen partition keys drive cost and latency up.

Security. Treat embeddings as sensitive derivatives of content. Apply the same RBAC, private networking, and encryption controls you use for the source documents. Never assume “it’s just floats” and skip tenant isolation in filters.

---

Exam Scenario Checklist

If a question describes storing product manuals and answering support questions with grounded GPT responses, look for: container vector embedding policy, DiskANN (or appropriate index), VectorDistance + TOP, same-item storage of text + vector, and exclusion of the vector path from the default index. If the scenario emphasizes exact matches on tiny datasets, flat may appear as the correct index choice. If it emphasizes continuous ingest at scale with low latency ANN, DiskANN is the signal answer.