16.1 OCI Logging: Log Groups, Service Logs, Custom Logs & the Unified Monitoring Agent

Why Centralized Logging Matters

When an incident happens, architects need to answer questions like "which client called this API?", "what did the load balancer return?", and "what did the application log just before it failed?". OCI Logging is a fully managed, highly scalable service that gives you a single place to enable, manage, and search the logs from OCI resources, applications, other clouds, and on-premises systems.

The Three Kinds of Logs

Logs and Log Groups

• A log is a first-class OCI resource with an OCID. For example, enabling flow logs on a subnet creates a dedicated log.
• A log group is a logical container for logs in a compartment. You must create one before you enable a log. Use log groups to apply IAM policies and to search related logs together, such as one log group per application or environment.
• Moving a log group to another compartment moves all of its logs with it.

Logs are encrypted in flight during ingestion, encrypted at rest, and encrypted when archived. Only UTF-8 text is supported.

Retention

For service and custom logs, you choose the retention period in 30-day increments up to 180 days. For longer retention, such as compliance archives kept for years, use Connector Hub to copy logs to Object Storage (Section 16.3).

Service Logs and Log Categories

Each service defines its own log categories, which are unrelated to other services' categories:

To enable a service log on a resource, you need permission on the log group and the update permission on the resource, which for many resources comes with the use verb. For example, users who can use buckets in a compartment can enable logging on those buckets.

Object Storage Logs Versus Audit

The Audit service records bucket-level API calls, such as creating a bucket or changing its visibility, but not object-level operations such as GetObject or PutObject. To answer "who downloaded this file?", enable the bucket's read log category, and its write category for uploads.

Custom Logs

Unified Monitoring Agent

The Unified Monitoring Agent is a fluentd-based agent that runs on OCI instances, delivered as the Oracle Cloud Agent plugin called Custom Logs Monitoring, which must be enabled on the instance. It works on both VM and bare metal instances.

An agent configuration tells the agent what to collect:

1. Host groups: the dynamic groups (or groups) of instances that use the configuration.
2. Log inputs: Log Directory (Tail) file paths, such as /var/log/app/*.log, or Windows Event Logs.
3. Parser: how to turn lines into structured fields, for example none, JSON, regular expression, multiline, syslog, or Apache formats.
4. Destination: the custom log that receives the entries.

Any log field longer than 10,000 characters is truncated during ingestion. Agent configuration updates are detected and loaded automatically. The same agent can also collect custom metrics, such as Prometheus endpoints, for Monitoring.

IAM for Agents

The instances must be in a dynamic group, and a policy must let that group get configurations and send logs:

allow dynamic-group app-fleet to use log-content in compartment apps

With use log-content, the instances can download their configuration, push logs, and search logs. Readers who only need to view logs get read log-content, which covers service and custom logs alike.

The Logging Ingestion API

Applications, other clouds, and on-premises systems can send entries directly with the PutLogs operation of the Logging Ingestion API, which uses an ingestion.logging.<region> endpoint. This is useful for containers, serverless code, or systems where installing the agent is not possible.

Logging APIs

• Logging Management API: create and manage logs, log groups, and agent configurations.
• Logging Ingestion API: send custom log entries (PutLogs).
• Logging Search API: search logs programmatically (Section 16.2).

Design Example: A Three-Tier Application

1. Create a log group per environment, such as prod-app-logs.
2. Enable Load Balancer access and error logs, API Gateway access and execution logs, and VCN flow logs on the application subnets.
3. Enable the Custom Logs Monitoring plugin on the application instances, add them to a dynamic group, and create an agent configuration that tails /var/log/app/*.log with a JSON parser.
4. Enable Object Storage write logs on the upload bucket.
5. Use Connector Hub to archive everything to Object Storage for one year, and to raise alarms on error patterns.