7.1 OCI Streaming, OCI Queue & Messaging Patterns for Microservices

5.3 OCI Streaming, OCI Queue & Messaging Patterns for Microservices

Enterprise Messaging Paradigms: Streams vs. Queues

Distributed enterprise architectures depend heavily on asynchronous messaging to decouple microservices, absorb unpredictable traffic surges, and guarantee data durability. However, cloud architects frequently conflate two fundamentally different messaging paradigms:

1. Event Streaming (Log-Centric Pub/Sub): An immutable, distributed, append-only log where continuous sequences of events are written to partitioned streams. Messages are retained independently of consumer activity, enabling multiple independent consumer applications to read, track progress, and replay event history.
2. Message Queuing (Point-to-Point Task Distribution): A transient buffer designed to decouple task producers from competing consumer workers. Messages are processed by whichever worker is available and are permanently deleted once acknowledged, with built-in mechanics for retries and dead-letter handling.

On Oracle Cloud Infrastructure, OCI Streaming Service (OSS) addresses the event streaming model, while OCI Queue provides the point-to-point task queue model.

---

OCI Streaming Service (OSS) Architecture

OCI Streaming is a fully managed, scalable, durable, and highly available real-time event ingestion and streaming platform. It provides a serverless alternative to deploying, patching, and managing self-hosted Apache Kafka clusters.

Stream Pool (KMS Encryption, Private VCN Endpoint, Kafka Settings)
  └── Stream: OrdersStream (Retention: 7 Days)
        ├── Partition 0 ──[Offset 0, 1, 2, 3...] ──> Consumer A (Group 1)
        ├── Partition 1 ──[Offset 0, 1, 2, 3...] ──> Consumer B (Group 1)
        └── Partition 2 ──[Offset 0, 1, 2, 3...] ──> Consumer C (Group 1) & Consumer X (Group 2)

Core Structural Components

• Stream Pools: A logical grouping and administrative boundary for multiple streams. Stream pools define shared security configurations, including Customer-Managed Encryption Keys (CMEK) via OCI Vault, private endpoint VCN attachment, and Kafka connection settings.
• Streams: A partitioned log of messages, created within a stream pool. Retention is set at creation between 24 hours and 7 days and cannot be changed later.
• Partitions: The fundamental unit of scalability, throughput, and parallelism in OCI Streaming. Sequential message ordering is strictly guaranteed within a single partition, but ordering is not guaranteed across different partitions.
• Message Keys: When producing a message, the producer specifies an optional message key. OCI Streaming hashes the key to assign the message to a specific partition. Messages with identical keys are guaranteed to land on the same partition in sequential order.
• Consumer Groups: A coordinated group of consumer instances that divide stream partitions among themselves for parallel message consumption. Each partition is allocated to exactly one consumer within the group. Consumers commit their read offsets back to OCI Streaming, allowing the group to resume processing seamlessly without duplicating records if an instance fails.
• Replayability: Reading a message does not delete it. Messages persist until the retention window expires. A consumer group can reset its offset to rewind and replay historical data, and distinct consumer groups can consume the identical stream independently at their own pace.

---

Limits, Sizing & Apache Kafka Compatibility

OCI Streaming publishes these limits (Oracle's "Limits on Streaming Resources" page):

Sizing Partitions

Because the partition count is fixed when the stream is created, size for peak load plus growth:

$$\text{Partitions} \ge \left\lceil \frac{\text{Peak write MB/s}}{1\ \text{MB/s per partition}} \right\rceil$$

Worked example: an IoT platform writes 10 MB/s at peak and expects 50% growth within a year. It needs at least 10 partitions today and 15 for next year's peak, so creating the stream with 15 partitions avoids recreating it later. Each consumer group reads every partition independently, subject to 5 GET requests per second per partition for that group, so adding a second consuming application means adding a consumer group rather than more partitions.

Apache Kafka Compatibility

Streaming is compatible with most Kafka APIs, so many existing Kafka applications can move without code changes. Oracle lists support for the Kafka Producer, Consumer, Connect, Admin, and Group Management APIs (v0.10 and later). Oracle also lists features that are not implemented:

• Kafka Streams
• Log compaction
• Transactions
• Dynamic partition addition
• Idempotent production

Connection details:

• Bootstrap server: the regional endpoint on port 9092, for example streaming.us-phoenix-1.oci.oraclecloud.com:9092.
• Authentication: SASL_SSL with the SASL/PLAIN mechanism. The username has the format {tenancyName}/{username}/{streamPoolId}, and the password is an auth token. The Java SDK can also authenticate with instance principals.
• Durability: Streaming supports only lossless Kafka configurations. Data is replicated three ways, and producers receive an acknowledgment only after at least two replicas are in sync.
• Kafka Connect: connectors can move data between Streaming and databases or object stores, for example through the JDBC connector.

---

OCI Queue Architecture & Worker Semantics

OCI Queue is a fully managed, serverless, point-to-point message queuing service built to handle high-throughput, bursty asynchronous task workloads without requiring partition sizing or cluster management.

Producer ──> OCI Queue ──> Visibility Timeout (30s) ──> Worker Pool (Competing Consumers)
                                │                              │
                       Processing Succeeds             Processing Fails 3x
                                │                              │
                       DELETE Message                  Dead-Letter Queue (DLQ)

Key Architectural Features

• At-Least-Once Delivery: Guarantees that every message is delivered to a consumer at least once. Consumer workers must be designed to be idempotent.
• Competing Consumers: Unlike stream partitions where only one consumer in a group reads a partition, all workers in a consumer pool pull from the same queue concurrently. The first available worker claims the next message.
• Visibility Timeout: When a worker retrieves a message, OCI Queue hides it from other consumers for the visibility timeout (configurable at the queue from 1 second to 12 hours, default 30 seconds; consumers can set it per request and extend it while they work). During this interval, the message remains hidden from other workers. If the worker processes the task successfully, it invokes DeleteMessage using the unique message receipt handle. If the worker crashes or times out before deletion, the visibility timeout expires, and the message automatically reappears in the queue for another worker to process.
• Dead-Letter Queues (DLQ): When a malformed message ("poison pill") repeatedly causes worker exceptions, retrying indefinitely would exhaust compute capacity and block queues. OCI Queue lets you set maximum delivery attempts from 1 to 20 at the queue level. A message delivered more times than that without being deleted moves to the queue's dead letter queue (DLQ) for investigation.
• Retention and Size: Messages are kept until they are deleted or until the retention period expires (configurable from 10 seconds to 7 days, default 1 day). Messages can be up to 256 KB.
• Channels and Consumer Groups: Channels are on-demand destinations inside a queue, so producers can publish to a specific channel and consumers can read from it (up to 256 channels per queue). Consumer groups let one queue fan out messages to several independent consuming applications.
• Protocols: OCI Queue can be used through its REST API or the industry-standard STOMP protocol, and Connector Hub can move queue messages to Functions, Notifications, Object Storage, or Streaming.

---

Decision Framework: OCI Streaming vs. OCI Queue

Selecting between OCI Streaming and OCI Queue is a foundational architectural choice on the 1Z0-997-26 examination: