7.2 Event-Based & Real-Time Data Integration Design
Key Takeaways
- Event Hubs is for high-throughput telemetry ingestion, distinct from Event Grid (discrete pub/sub events) and IoT Hub (adds device management/command-and-control).
- Dedicated and Premium Event Hubs tiers guarantee isolated throughput at large scale; Standard tier uses shared throughput units.
- Event Hubs Capture provides a durable, unmodified copy of raw events for audit/replay while Stream Analytics processes the live stream in parallel.
- Stream Analytics window types map directly to requirement language: tumbling (fixed, non-overlapping), hopping (overlapping rolling), sliding (event in/out triggers), session (activity-gap grouping).
- A single Stream Analytics job can write to multiple outputs (Power BI, SQL, Cosmos DB, Data Lake) at once — recognize fan-out scenarios instead of over-designing separate pipelines.
Why Real-Time Integration Design Is Tested Separately
Batch pipelines like the ones in the previous section run on a schedule and tolerate minutes or hours of latency. A growing share of AZ-305 scenarios instead describe telemetry, clickstreams, financial ticks, or IoT sensor readings that must be ingested continuously and acted on within seconds. The exam expects you to recognize when a requirement is really asking for event-based, real-time data integration rather than scheduled batch movement, and then to select the right ingestion service and the right stream-processing engine together — picking only one half of that pair is a common wrong answer on practice questions.
Ingestion Layer: Azure Event Hubs
Azure Event Hubs is a big-data streaming platform and event ingestion service built to receive and process millions of events per second from many concurrent producers (devices, applications, or clickstreams). Key design vocabulary:
- Partitions — an Event Hub is divided into partitions so that multiple consumers can read in parallel; the partition count is fixed at creation time on the Standard tier and should be sized to match the maximum expected number of concurrent readers.
- Throughput Units (TU) on the Standard tier and Processing Units (PU) on the Premium tier / Capacity Units (CU) on the Dedicated tier define ingress and egress capacity; Dedicated is the design answer whenever a scenario needs guaranteed, isolated capacity at massive scale (millions of events/sec, single-tenant).
- Consumer Groups give each downstream application (for example, one for real-time dashboards and a separate one for long-term storage) its own independent view of the stream, so one slow consumer never blocks another.
- Event Hubs Capture automatically writes the raw incoming stream to Azure Blob Storage or Azure Data Lake Storage on a time or size interval — the correct answer whenever a scenario needs both real-time processing and a durable, unmodified copy of the raw events for later replay or compliance.
- Kafka endpoint compatibility lets existing Apache Kafka producers/consumers connect to Event Hubs without code changes, which matters for migration scenarios from self-managed Kafka clusters.
Do not confuse Event Hubs with Azure Event Grid (a lightweight, reactive pub/sub service for discrete system events, like a blob-created notification, not continuous telemetry streams) or Azure IoT Hub (adds bidirectional device management, device twins, and per-device security on top of similar ingestion — the right choice specifically when the scenario needs to manage or send commands to devices, not just receive their data).
Processing Layer: Azure Stream Analytics
Azure Stream Analytics is a fully managed, serverless real-time analytics service that reads from a streaming input (Event Hubs, IoT Hub, or Blob storage), runs a SQL-like query written in Stream Analytics Query Language (SAQL), and writes results to an output sink. The windowing functions are consistently tested:
| Window Type | Behavior | Typical Use |
|---|---|---|
| Tumbling | Fixed-length, non-overlapping, contiguous | "Average temperature every 5 minutes" |
| Hopping | Fixed-length, can overlap (defined hop size) | Rolling 10-minute average recalculated every minute |
| Sliding | Emits output whenever an event enters or exits the window | Detecting exactly when a threshold is crossed |
| Session | Groups events separated by less than a timeout gap | Grouping a user's clickstream into "sessions" |
Stream Analytics can write to many outputs simultaneously from one job — a common exam pattern is Power BI for a live executive dashboard plus Azure SQL Database or Cosmos DB for operational storage plus Blob/Data Lake for archival, all from a single query with multiple INTO clauses.
Exam Scenario Walkthrough
A logistics company installs GPS and temperature sensors on 50,000 refrigerated delivery trucks. Requirements: ingest telemetry at massive scale with guaranteed dedicated capacity, retain a durable unmodified copy of every reading for regulatory audit, compute a rolling 5-minute average temperature per truck to detect spoilage risk in near real time, and push alerts to a live operations dashboard while also archiving all readings to a data lake. The design: an Event Hubs Dedicated tier cluster (guaranteed isolated capacity at that scale) with Event Hubs Capture enabled to Azure Data Lake Storage (durable raw copy for audit), and an Azure Stream Analytics job with a tumbling window query that computes the 5-minute rolling average and writes simultaneously to Power BI (dashboard) and Azure Data Lake (archival copy of aggregated results).
Takeaways
- Event Hubs is for high-throughput telemetry ingestion; Event Grid is for discrete pub/sub system events; IoT Hub adds device management on top of ingestion — pick based on what the scenario actually needs, not just "streaming data."
- Event Hubs Capture is the answer whenever a design needs both real-time processing and an unmodified durable copy of the raw stream.
- Dedicated (or Premium) tier Event Hubs is the answer for guaranteed, isolated throughput at very large scale; Standard tier throughput units are for smaller, shared workloads.
- Match the Stream Analytics window type to the requirement: tumbling for fixed non-overlapping intervals, hopping for overlapping rolling windows, sliding for threshold-crossing detection, session for activity-gap grouping.
- A single Stream Analytics job can fan out to multiple outputs at once — recognize this pattern instead of designing separate pipelines per destination.
A manufacturing company needs to ingest sensor telemetry from 200,000 factory devices at a guaranteed, isolated throughput level with no risk of noisy-neighbor contention from other tenants. Which Event Hubs tier should you recommend?
A design requires computing a rolling 10-minute average of stock price every 1 minute, where each 10-minute window can overlap with the previous one. Which Azure Stream Analytics window type fits this requirement?
A retail company needs to both run real-time fraud-detection queries on transaction events and retain an unmodified copy of every raw event for a 7-year compliance requirement. Which Event Hubs feature satisfies the durable raw-copy requirement?