5.2 Change Feed Processor for New/Updated Items

Change Feed as the AI Ingest Nervous System

Operational AI systems rarely load a static corpus once. Support articles, product catalogs, chat transcripts, and IoT telemetry keep changing. Azure Cosmos DB change feed gives you an ordered, per-partition log of inserts and updates so downstream code can react without scanning the whole container.

For AI-200, connect change feed to the vector story from the previous section: when an item is written or updated, a processor can call an embedding model, write contentVector back to the item (or to a derived container), and make that content immediately available to DiskANN similarity search. The same feed can fan out to Azure Functions, Azure Service Bus, Azure AI Search indexers, logging, or evaluation pipelines.

The change feed is not a full transaction log of every property-level audit event in all modes, and classic latest-version processing focuses on the current item shape after writes. Know that deletes and “all versions” behaviors depend on which change feed mode you enable; exam scenarios for AI ingest usually emphasize detecting new or updated items and running asynchronous work.

---

Push Model: Change Feed Processor

Microsoft generally recommends the change feed processor (often called the push model) when you want continuous processing. Your code registers a delegate that receives batches of changes. The library handles:

• Polling for new changes on an interval you configure
• Persisting checkpoints so a restart resumes correctly
• Distributing partitions/leases across multiple worker instances
• Retrying transient failures according to processor behavior

Core Components

The lease container can live in the same Cosmos DB account as the monitored container or in another account. Each lease tracks progress for a portion of the feed. At any moment, a given lease is owned by at most one instance. If you run more instances than leases, extras sit idle until work is redistributed. If an instance dies, another instance can take over its leases after the lease expires or is released.

Lifecycle Sketch

1. Create the lease container (often with /id as partition key) if it does not exist.
2. Build the processor against the monitored container: set instance name, lease container, optional start time or “from beginning,” and the change handler.
3. Start the processor; it acquires leases and begins invoking your handler.
4. On success, the library checkpoints so those changes are not redelivered indefinitely under normal processing.
5. Scale out by adding instances with the same processor name and lease container; the library rebalances leases.

Start position matters only before leases are initialized. Options such as starting from the beginning of the container’s lifetime or from a timestamp apply when the lease set is first created. After leases exist, changing those start options typically has no effect—progress is driven by lease checkpoints.

Azure Functions Integration

A common exam-friendly pattern is the Cosmos DB trigger for Azure Functions. The trigger is built on change feed processor concepts: you point at a monitored collection/container and a lease collection. Each function invocation receives new or updated documents. That is an ideal place to:

• Generate embeddings and patch vectors
• Write enrichment fields
• Send messages for further fan-out

Keep functions idempotent where possible. At-least-once delivery semantics mean a batch might be retried after a crash; embedding the same text twice should yield a safe upsert, not duplicate side effects without keys.

---

Pull Model Versus Push Model

The pull model lets your client request change feed pages explicitly using continuation tokens and feed ranges. You control pace, parallelism, and persistence of bookmarks yourself.

You cannot convert a pull continuation token into a lease document or vice versa. Pick a model and stay consistent for that pipeline.

Use pull when you must tightly control batch size and schedule (for example a nightly job that drains changes into a training set) or when you need fine-grained feed-range reads. Use push / change feed processor for always-on AI enrichment workers.

---

AI Pipeline Fan-Out Use Cases

Change feed shines when one write should trigger multiple asynchronous AI-related actions without slowing the API that inserted the item.

1. Embedding and Vector Upsert

Handler reads content from each new/updated item → calls Azure OpenAI embeddings → upserts contentVector (and maybe embeddedAt). DiskANN indexes pick up the vector for later VectorDistance queries. This keeps write latency low for clients while search quality stays current.

2. Secondary Index / Search Projection

Fan out to Azure AI Search or another read-optimized store when you need hybrid ranking features beyond what the current Cosmos query surface provides. The change feed item is the source of truth event.

3. Evaluation, Safety, and Telemetry

Push changed content to content-safety screening, quality evaluation queues, or analytics. Failures in those branches should not corrupt lease progress for the primary embedding path—often solved with separate processors (different processor names / lease prefixes) so one pipeline’s poison message does not block another.

4. Notification and Orchestration

Publish to Service Bus or Event Grid so multi-step orchestrations (chunking → embed → evaluate → notify) run as independent consumers. Cosmos DB remains the system of record; the feed is the trigger.

Lease Prefixes for Parallel Pipelines

If two deployment units share one lease container, assign each a distinct lease prefix. Without that separation, processors can interfere by competing for the same lease documents. Separate lease containers also work and are easier to reason about in many designs.

---

Design and Exam Pitfalls

• Hot partitions in the monitored container still affect how quickly changes appear per partition; fix partition key design rather than only adding workers.
• Handler time budgets: long embedding calls may need concurrency limits, batching, or queue offload so lease processing stays healthy.
• Exactly-once illusions: design upserts and downstream writes to tolerate duplicates.
• Estimators: change feed estimator APIs help measure lag (how far behind processors are)—useful operationally when RU throttling or slow embedding endpoints cause backlog.
• Security: processors need least-privilege access to monitored and lease containers plus whatever embedding or messaging endpoints they call; store secrets in Key Vault or managed identity patterns.

---

Putting It Together With Vector Search

A complete Cosmos-centric RAG ingest path for the exam:

1. App upserts chunk documents into the monitored container (text + metadata, vector optional at first).
2. Change feed processor (push) leases partitions and receives new/updated items.
3. Worker embeds text and writes vectors under the policy path; DiskANN maintains ANN searchability.
4. Query path embeds the user question and runs filtered VectorDistance top-k retrieval.
5. Optional second processor with another lease prefix fans out to audit or search projection without blocking embeddings.

If a question asks how to detect new or updated Cosmos DB items and run AI enrichment at scale with automatic checkpointing, the answer centers on the change feed processor, a lease container, and often an Azure Functions Cosmos DB trigger. If it stresses manual pacing and continuation tokens, look for the pull model.