15.1 OCI Monitoring: Metrics, Namespaces, Custom Metrics & Monitoring Query Language
Key Takeaways
OCI Monitoring receives metrics as raw data points (timestamp-value pairs) with dimensions and metadata, from OCI resources or as custom metrics.
An MQL query must specify a metric, an interval, and a statistic, such as
CpuUtilization[1m].mean().With a 1-minute interval, a metric query returns at most 7 days of data, while 1-hour and 1-day intervals return up to 90 days.
Custom metric namespaces cannot use the reserved prefixes
oci_ororacle_.Custom metrics can be posted as often as every second, but the minimum aggregation interval is one minute.
How Monitoring Works
The OCI Monitoring service has two features: Metrics, which collect and query measurements of health, capacity, and performance, and Alarms, which notify you when metrics meet a trigger (Section 15.2). Metric and alarm data is available from the Console, CLI, and API.
Metrics come from three sources:
- Resource (service) metrics that OCI services post automatically. For example, Compute posts
CpuUtilizationfor monitoring-enabled instances in theoci_computeagentnamespace through the Oracle Cloud Agent. - Custom metrics that your applications publish with the Monitoring API, or that the Unified Monitoring Agent collects through agent configurations, such as Prometheus-format metrics exposed on an HTTP endpoint of a VM.
- Connector Hub, which can move data from Logging into Monitoring as metrics (Section 16.3).
Core Concepts
| Concept | Meaning | Example |
|---|---|---|
| Data point | A timestamp-value pair, raw (as posted) or aggregated | 2022-05-10T22:19:00Z, 10.4 |
| Metric namespace | The service or application that emits the metric | oci_computeagent, oci_objectstorage, oci_redis |
| Dimension | A name-value qualifier used to filter or group | resourceId, availabilityDomain, faultDomain |
| Metadata | Reference information such as the unit | bytes for DiskBytesRead |
| Metric stream | Aggregated data for one metric and one set of dimension values; one line on a chart | CPU of one instance |
| Frequency | How often raw data points are posted; most service metrics post every 60 seconds | 1 per minute |
| Interval | The window over which raw points are aggregated | [1m], [5m], [1h] |
| Statistic | The aggregation function applied in each interval | mean(), max(), percentile(0.9) |
| Resolution | How often the aggregation windows shift; for metric queries it defaults to the interval | 1m |
| Resource group | An optional custom string on a custom metric, used to filter or aggregate; one per metric | divisionX |
Metrics are not OCI resources and have no OCIDs. Access is controlled with IAM policies on the metrics resource type.
Monitoring Query Language (MQL)
An MQL expression always contains a metric, an interval, and a statistic, and optionally dimension filters, grouping functions, and a predicate:
CpuUtilization[1m]{availabilityDomain = "VeBZ:PHX-AD-1"}.groupBy(poolId).percentile(0.9) > 85
Reading it from left to right: take CpuUtilization, aggregate it every 1 minute, keep only streams in one availability domain, group the streams by instance pool, compute the 90th percentile, and keep values above 85.
Intervals and Time Ranges
Valid intervals are 1m to 60m, 1h to 24h, and 1d. The interval determines the default resolution and therefore the maximum time range a metric query returns:
| Interval | Maximum time range returned |
|---|---|
| 1 minute | 7 days |
| 5 minutes | 30 days |
| 1 hour | 90 days |
| 1 day | 90 days |
Pick the alarm interval based on how often the metric is emitted; a metric emitted every five minutes needs an interval of at least 5 minutes.
Statistics
| Statistic | Returns |
|---|---|
mean() / avg() | Sum divided by count |
max(), min() | Highest or lowest value in the interval |
sum() | All values added together |
count() | Number of observations |
percentile(p) | Estimated percentile, with p between 0 and 1, for example percentile(0.95) |
rate() | Per-interval average rate of change, per second |
increment() | Per-interval change |
first(), last() | Earliest or latest value in the interval |
absent() | True when the metric is absent for the entire interval |
Each query uses one statistic per metric. For a throughput rate, write NetworksBytesIn[1m].rate(), not a chain of statistics such as .sum().rate().
Filters, Fuzzy Matching, and Grouping
- Dimension filters go in braces:
{resourceId = "ocid1.instance..."}. - Fuzzy matching uses
=~with*as a wildcard and|for alternatives:{resourceDisplayName =~ "web-*|api-*"}. groupBy(dimension)aggregates streams that share a dimension value, such as one line per instance pool.grouping()combines all streams into a single value, such as total requests across all resources.
Predicates, Arithmetic, and Joins
- Predicates:
>,>=,==,!=,<,<=,in (low, high)(between, inclusive),not in (low, high), andabsent(). - Arithmetic:
+,-,*,/, and%. For example,100 - CpuUtilization[1m].mean()gives idle CPU, andTotalRequestLatency[1m].mean() / 1000converts milliseconds to seconds. - Joins:
&&(AND) and||(OR) combine queries. For example,ServiceConnectorHubErrors[1m].count() > 1 && ServiceConnectorHubErrors[1m].mean() > 0.5is true only when both conditions hold.
Custom Metrics
Publish business or application metrics, such as orders per minute or failed payments, with the PostMetricData API or the CLI command oci monitoring metric-data post. Key rules:
- Posting uses the
telemetry-ingestionendpoint, while queries use thetelemetryendpoint. - Do not start the namespace with
oci_ororacle_. - Data points can be posted as often as every second, but the minimum aggregation interval is one minute.
- Timestamps must be near the current time: less than two hours in the past and less than 10 minutes in the future.
- Publishers need an IAM policy such as
allow group app-publishers to use metrics in compartment apps where target.metrics.namespace = 'orders_app'.
The Unified Monitoring Agent with agent configurations is an alternative that does not need custom API code, for example to scrape Prometheus endpoints on VMs.
Limits That Affect Design
- Metric definitions and alarm history are stored for 90 days.
- A metric query returns at most 100,000 data points, so long time ranges need larger intervals.
- An alarm tracks up to 200,000 metric streams.
Visualizing Metrics
The Service Metrics page shows default charts for each namespace, the Metrics Explorer builds custom charts from MQL, and dashboards combine widgets for operations teams. Each chart line is one metric stream unless you aggregate the streams.
An operations team wants one line per instance pool showing the 95th percentile CPU of the pool's instances in a single availability domain. Which MQL expression produces this?
CpuUtilization[1m]{availabilityDomain = "VeBZ:PHX-AD-1"}.grouping().percentile(0.95)
CpuUtilization[1m]{availabilityDomain = "VeBZ:PHX-AD-1"}.groupBy(poolId).percentile(0.95)
CpuUtilization[1m]{availabilityDomain = "VeBZ:PHX-AD-1"}.groupBy(poolId).percentile(95)
CpuUtilization[1m]{availabilityDomain = "VeBZ:PHX-AD-1"}.groupBy(poolId).max().percentile(0.95)
A developer publishes a custom metric with the namespace oci_orders through PostMetricData and the request is rejected. What is the most likely cause?
Custom metrics must be published through the telemetry query endpoint
Custom metrics can be posted at most once per minute per metric stream
The oci_ prefix is reserved and cannot be used for custom namespaces
Custom metrics require a resource group on every posted data point
An analyst needs a chart covering the last 60 days of CPU utilization for a fleet. Which interval should the query use?
1 minute, for the most detailed chart
5 minutes, as a compromise between detail and range
Any interval, because Monitoring returns all stored data regardless of interval
1 hour or 1 day, because only those intervals return more than 30 days of data
Sections you finish are checked off in the contents.