9.2 Topics, Subscriptions & Dead-Letter Queues

From Point-to-Point Queues to Pub/Sub Topics

A queue delivers each message to a single competing consumer. A topic accepts a published message and copies it to each matching subscription. Each subscription behaves like a virtual queue: independent consumers, independent delivery counts, and their own dead-letter subqueues.

For AI systems, topics shine when one domain event should trigger multiple reactions:

• document.processed → update search index, notify the user, write audit logs, kick off evaluation metrics.
• model.deployed → warm caches, refresh prompt configurations, alert MLOps.
• safety.flagged → quarantine content, notify compliance, open a human review ticket.

Producers publish once; you add or remove subscribers without changing the publisher—critical as AI platforms grow new side effects around the same core events.

Topics, Subscriptions, and Filters

When a message is sent to a topic, Service Bus evaluates each subscription's filter rules:

Filtered subscriptions only receive copies that match. That keeps embedding workers from wasting cycles on evaluation-only messages and prevents noisy fan-out.

Rule actions can enrich properties as messages enter a subscription (for example, set routedTo = 'embed-workers'). Use filters for routing decisions; keep business logic in workers.

Design tips for AI pub/sub:

• Put routing dimensions in application properties (jobType, tenantId, modelName, region) rather than burying them only inside a JSON body—filters evaluate properties efficiently.
• Prefer a small set of stable event types with rich properties over dozens of topics that are hard to govern.
• Remember each matching subscription gets its own copy; Completing on one subscription does not remove copies from others.

Dead-Letter Queues (DLQ) in Depth

Every queue and subscription has a dead-letter subqueue. Messages move to the DLQ when:

1. Max delivery count is exceeded after repeated Abandon/lock-expiry cycles (classic poison message).
2. The message expires (TTL) and dead-lettering on expiration is enabled.
3. A receiver explicitly calls DeadLetter with a reason and error description.
4. Certain filter/forwarding failures occur (depending on configuration).

Dead-lettered messages retain headers such as DeadLetterReason and DeadLetterErrorDescription. For AI jobs, set explicit reasons like UnsupportedMimeType, SchemaValidationFailed, or ModelEndpointPermanent404 so on-call engineers and reclaim Functions can automate triage.

Operational Pattern: DLQ Reclaim for AI Pipelines

Healthy AI platforms treat the DLQ as a first-class backlog:

1. Detect — alert when DLQ depth > 0 or growth rate spikes (Application Insights metric / Azure Monitor).
2. Inspect — peek DLQ messages; read reason, correlation ID, blob URI, and delivery count.
3. Classify — transient misconfiguration (wrong model deployment name) vs permanently bad payload (corrupt PDF).
4. Remediate — fix config, repair or replace blob content, then resubmit a corrected message to the main queue/topic.
5. Complete the DLQ message after successful reclaim so it does not get processed twice from the DLQ path.

Never silently purge DLQs in production AI systems that handle customer documents—you may discard the only pointer to unpaid or incomplete work. If you must delete, archive the body and properties to cold storage first.

Max Delivery Count and Poison Embeddings

Suppose an embedding worker throws because a chunking Function wrote empty text. Each Peek-Lock attempt Abandons or lets the lock expire; delivery count increments. When it hits max delivery count (commonly 10), Service Bus dead-letters the message automatically. That protects the fleet from infinite retry storms on poison data.

Contrast with intentional Abandon for throttling (HTTP 429 from Azure OpenAI): you want retries, so ensure transient errors do not immediately DeadLetter. Use:

• Abandon / lock expiry for transient faults.
• Scheduled messages or exponential backoff via delayed re-enqueue for smarter retry timing.
• Explicit DeadLetter for permanent business failures.

Combining Queues and Topics in One AI Architecture

A practical topology for exam-style scenarios:

1. Ingest queue — API enqueues ingest commands; competing consumers perform OCR/chunking (point-to-point load distribution).
2. Pipeline topic — on success, publish stage.completed with properties { stage: 'chunked', documentId, tenantId }.
3. Subscriptions — embed-sub filters stage = 'chunked'; notify-sub filters all stages for UX updates; metrics-sub takes everything for quality dashboards.
4. DLQ per subscription — embed failures do not block notify/metrics copies.

This matches the official skill: queue and process back-end operations with Service Bus, including messages, topics, subscriptions, and dead-letter queue handling.

Forwarding and Auto-Forward (Awareness)

Subscriptions can auto-forward to another queue or topic to chain stages without custom glue code. Use forwarding for simple hops; use Explicit worker Complete logic when the stage needs AI compute validation before advancing.

Exam Watch-Outs

• Topics = one publish, many subscription copies; queues = one consumer wins each message.
• Filters select which subscriptions receive a copy; they do not rewrite the publisher’s intent retroactively for non-matching subscriptions.
• DLQ is per queue/subscription—fixing one consumer’s DLQ does not clear others.
• Explicit DeadLetter reasons are exam-friendly signals of correct poison handling.
• Fan-out side effects (search index + notify + audit) are a topic/subscription story, not a single queue with one worker doing everything synchronously.