16.3 Connector Hub: Moving Logs, Metrics, Queue Messages & Streams Between Services

What Connector Hub Does

Connector Hub, formerly Service Connector Hub, is a cloud message bus for moving data between OCI services. Instead of writing and running code to copy logs to a bucket or forward metrics to a SIEM, you define a connector:

• Source: where the data comes from.
• Tasks (optional): a log filter task to select log entries, or a Functions task to transform data.
• Target: where the data goes.

Connectors run continuously, and data moves in a sequence of move operations. Delivery speed depends on the source, task, target, and their service limits. For example, a Logging-to-Notifications move takes up to a few minutes, and a Streaming source with a Functions task and Object Storage target can take up to about 17 minutes depending on batch settings.

Sources and Targets

Targets behave differently: Functions processes data; Log Analytics, Monitoring, Object Storage, and Streaming store it; and Notifications delivers it to topic subscriptions.

Source Details

• Logging: select one or more logs or log groups, including the _Audit log group. The Logging source has a 24-hour retention period in Connector Hub.
• Monitoring: select metric namespaces in one or more compartments; metric data points are aggregated before delivery. Retention is also 24 hours.
• Queue: messages are written to the target and then automatically deleted from the queue.
• Streaming: retention is the stream's own retention period, which you define.

Tasks

• The log filter task is supported only for a Logging source. It uses the Logging query syntax, a search scope plus a where condition, such as entries whose data.statusCode is 500 or higher. It does not use JMESPath or SQL.
• A Functions task sends batches to a function for custom processing, such as masking personal data, enriching records, or converting formats. The task returns processed data to Connector Hub, which writes it to the target. A Functions task is not supported for Queue or Streaming sources with a Notifications target.

Delivery, Retries, and Batching

• Connector Hub follows at-least-once delivery: each batch is delivered at least once, so targets may see duplicates and should handle them.
• If a move fails, the connector retries it and does not move later batches until the retry succeeds.
• If failures continue beyond the source's retention period, that batch is not delivered. For Logging, data generated more than 24 hours earlier can be skipped after a long outage.
• The Notifications target accepts messages up to 128 KB; larger messages are dropped.
• Batch settings exist for a Functions task, Functions target, and Object Storage target. You can set a batch size and a batch time, and whichever limit is reached first triggers the flush. For example, with 1 MB and 10 seconds, a batch is sent when 1 MB has accumulated or 10 seconds have passed.

Failing and Updated Connectors

• A connector that keeps failing with 404 errors, or 502 errors from a function, triggers a warning announcement after four consecutive days and is automatically deactivated after seven days. Fix the configuration and reactivate it, and enable connector logs to see the data flow.
• Updating a connector's source, task, or target resets it internally, so it may not resume from its previous position and may reprocess data. Oracle recommends creating a new connector when the source or target must change.

IAM for Connectors

A connector acts as its own principal (request.principal.type = 'serviceconnector'). When you create or edit a connector in the Console, default policies are offered for its source, task, and target. For example, an Object Storage target gets:

Allow any-user to manage objects in compartment id <bucket_compartment_ocid> where all {request.principal.type='serviceconnector', target.bucket.name='<bucket_name>', request.principal.compartment.id='<connector_compartment_ocid>'}

For narrower access, create a dynamic group that matches connectors, such as All {resource.type = 'serviceconnector', resource.compartment.id = '<compartment_ocid>'}, and write policies for that dynamic group. A Logging source has no default policy: the person creating the connector needs read access to the logs. Automatically created policies remain after a connector is deleted, so delete them as part of cleanup.

Common Designs

Design Example: Alerting on IAM Policy Changes

A security team wants an email whenever someone creates, updates, or deletes an IAM policy:

1. Create a connector with the Logging source set to the _Audit log group of the relevant compartments.
2. Add a log filter task whose where condition matches the policy event types, for example a type containing identitycontrolplane and policy.
3. Set the target to a Notifications topic with an email subscription for the security team.
4. Accept the default policy offered for the topic.

To count changes over time instead of sending one email per event, send the filtered entries to a Monitoring target and create an alarm on the resulting metric.