9.4 Event Grid Retries & Event-Driven Workflows

Reliable Delivery: Retries, Backoff, and Dead-Lettering

Event Grid’s push model only works if handlers are reachable and healthy. When a destination returns a failure or times out, Event Grid retries delivery using an exponential backoff policy for up to a maximum event time-to-live / retry period (configurable on the subscription; defaults are generous—often on the order of 24 hours unless you tighten them).

Understanding retries is essential for AI workflows: a Function that enqueues embedding work might fail because Service Bus is briefly throttling, or a webhook might time out while a cold start occurs. Retries bridge those transient gaps—but they also mean your handler can see the same event more than once.

What Counts as Success or Failure?

For HTTP-style destinations:

• Success: destination returns 200–299 within the timeout.
• Retryable failure: timeouts, 503, and many 5xx responses → Event Grid schedules another attempt.
• Non-retryable / terminal scenarios: certain 400-class failures may stop retries depending on policy and error type; misconfigured endpoints that fail validation never become healthy subscriptions.

Azure Functions with Event Grid triggers integrate with this model: throw/fail the invocation for transient issues you want retried; succeed only after side effects are safely committed or durably queued.

Dead-Letter Destinations

If Event Grid exhausts retries (or the event expires), it can write the event to a dead-letter destination—commonly a Storage account blob container configured on the event subscription, with a managed identity or SAS granting Event Grid permission to write.

Dead-letter blobs typically include the original event and metadata about why delivery failed. For AI operations teams:

1. Alert on dead-letter container growth.
2. Inspect failing payloads (documentId, model name, tenant).
3. Fix the handler (bug, auth, wrong queue name).
4. Replay by re-publishing to the custom topic or by a reclaim script that re-invokes the pipeline.

Compare with Service Bus DLQ: both isolate poison/failed work, but Event Grid dead-letters delivery failures to subscribers, whereas Service Bus DLQ isolates consumer processing failures of queued messages. Mature AI platforms use both at their respective layers.

Idempotent Handlers Are Mandatory

Because retries and competing architecture layers create duplicates:

• Use event id + handler name as an idempotency key in Cosmos DB/Redis before starting expensive GPU work.
• Make “enqueue Service Bus message” idempotent with Service Bus duplicate detection (MessageId = event id) where appropriate.
• Ensure “update document status to Embedded” is a conditional write, not a blind append of duplicate rows.

Composing Event-Driven AI Workflows

The official skill emphasizes implementing event-driven workflows with Event Grid—including filters, custom events, and retries. A reference workflow that often appears in solution designs:

Scenario: Secure Document Q&A Ingestion

1. User uploads PDF to inbox container.
2. System topic on the storage account emits BlobCreated.
3. Event Grid subscription filters subject ends with .pdf and data.contentType for application/pdf.
4. Function A validates virus scan status and tenant path; on success, sends a Service Bus queue message (ingest job) and returns 200. On transient Service Bus errors, Function fails → Event Grid retries.
5. Service Bus workers run Document Intelligence, chunking, embeddings (Peek-Lock, sessions per document).
6. On success, worker publishes custom CloudEvent Document.Embedded to a custom topic.
7. Subscription B (filter type = embedded) updates Azure AI Search and returns 200.
8. Subscription C (filter tenant prefix) pushes a SignalR/user notification.
9. Subscription D (advanced filter data.chunkCount NumberGreaterThan 500) triggers an extra quality-evaluation job.
10. If Subscription B’s search endpoint is down beyond the retry window, the event lands in Event Grid’s dead-letter container for replay after search recovers.

This design separates reaction (Event Grid) from heavy work (Service Bus) and uses filters so each subscriber stays focused.

Configuring Retry Policy Thoughtfully for AI

• Short-lived glue Functions: default retries are fine; ensure cold starts still finish within Event Grid’s per-attempt timeout or use premium plans.
• Handlers that call slow model APIs directly inside the Event Grid request path are a design smell—offload to Service Bus/queue-triggered Functions so Event Grid only needs a quick enqueue success.
• Tighten retry TTL for events that must be ignored if stale (for example, “cancel generation” may not be useful 24 hours later); lengthen for “billing meter” style events that must eventually land.
• Always configure dead-lettering in production subscriptions—silent drop after retries hides AI pipeline outages.

Failure Modes to Recognize on the Exam

• Handler returns 200 before enqueuing work → Event Grid will not retry, but work may be lost if the process crashes after the 200. Commit then acknowledge (or outbox pattern).
• Handler takes minutes calling a model during Event Grid delivery → timeouts → repeated retries → duplicate partial work. Enqueue and finish quickly.
• No dead-letter destination configured → exhausted retries vanish from easy recovery.
• Filters too broad → storm of irrelevant invocations increases failure noise and cost.
• Filters too narrow → missed events look like “AI pipeline stuck” when Event Grid never delivered.

End-to-End Reliability Checklist

• Event Grid subscription: event types + subject/advanced filters aligned to AI domain properties.
• Retry policy + dead-letter container with monitoring.
• Handler idempotency using event ids.
• Bridge to Service Bus for durable back-end operations (queues/topics/DLQ from earlier sections).
• Custom events announce stage completion for fan-out without tight coupling.

Mastering this chapter means you can explain—not just name—when to queue with Service Bus, when to react with Event Grid, how filters shape delivery, and how retries/dead-lettering keep asynchronous AI pipelines trustworthy under partial failure.