6.1 OCI Generative AI Service: Models, Hosting & Dedicated Clusters

6.1 OCI Generative AI Service: Models, Hosting & Dedicated Clusters

Exam Tip: For the OCI AI Foundations Associate (1Z0-1122-26) exam, understand the operational boundary between On-Demand Serving and Dedicated AI Clusters. Remember that custom model fine-tuning (using LoRA or T-Few) and private endpoint hosting strictly require a dedicated AI cluster. Learn the portfolio by model family and task category rather than by version number: chat models (Cohere Command, Meta Llama, and other families Oracle adds over time), embedding models such as Cohere Embed, and rerank models. Remember that Cohere Command R / R+ carry a 128,000-token context window built for Retrieval-Augmented Generation (RAG) and tool use, and that Cohere Embed produces dense semantic vectors across 100+ languages. Oracle retires and replaces individual model versions continuously, so the exam rewards knowing what each category does, not which checkpoint shipped in a given year.

---

Overview of OCI Generative AI Service

The Oracle Cloud Infrastructure (OCI) Generative AI Service is a fully managed, enterprise-grade cloud service that integrates state-of-the-art Large Language Models (LLMs) into business workflows through unified Representational State Transfer (REST) Application Programming Interfaces (APIs) and Software Development Kits (SDKs). Organizations can leverage advanced generative artificial intelligence capabilities without procuring, configuring, clustering, or managing underlying Graphics Processing Unit (GPU) supercomputing hardware.

In enterprise deployments, data security and confidentiality are paramount. The OCI Generative AI service enforces strict governance boundaries:

• No Data Sharing: Customer prompts, context data, and generated responses are never shared with third-party model providers such as Cohere or Meta.
• Zero Model Retention: Base foundation models are never trained, retrained, or altered using customer prompts or inference data. Customer interactions remain completely ephemeral during standard inference.
• Tenancy Isolation: Fine-tuned custom models and their associated training weights reside exclusively within the customer's dedicated tenancy and compartment, secured by OCI Identity and Access Management (IAM) policies and encrypted both in transit (TLS 1.3) and at rest (AES-256).

+-----------------------------------------------------------------------------+
|                     OCI GENERATIVE AI SERVICE BOUNDARY                      |
|                                                                             |
|  Enterprise Client Applications (ERP, HCM, SCM, Custom Web / Mobile Apps)  |
|                                      │                                      |
|                       HTTPS / REST API / OCI SDKs                           |
|                                      ▼                                      |
|                 OCI Identity and Access Management (IAM)                    |
|                     Role-Based Access Control (RBAC)                        |
|                                      │                                      |
|         ┌────────────────────────────┴────────────────────────────┐         |
|         ▼                                                         ▼         |
|   ON-DEMAND SERVING                                     DEDICATED AI CLUSTER|
|   - Multi-tenant shared GPUs                            - Single-tenant GPU |
|   - Pay-per-token pricing                               - Guaranteed latency|
|   - Pre-trained models only                             - Fine-tuning & VCN |
+-----------------------------------------------------------------------------+

---

Foundation Model Portfolio: Chat, Embedding, and Rerank Models

The OCI Generative AI service curates foundation models from several industry-leading AI labs and runs them on high-performance Oracle Cloud Infrastructure hardware. Oracle groups the catalogue by task category rather than by vendor, and the individual models within each category are added, deprecated, and retired on a rolling basis — Oracle publishes the current list, plus each model's deprecation and retirement dates, on its Offered Pretrained Foundational Models documentation page.

[!IMPORTANT] Learn the categories, not the version numbers. Named checkpoints move fast: models that were the flagship offering a year ago are frequently marked deprecated in the current catalogue. Exam questions at the Foundations level test whether you can pick the right kind of model for a scenario — chat versus embedding versus rerank — and which hosting mode that requires. Check Oracle's documentation for the live model list before quoting a specific model name in production work.

1. Chat Models

Chat models take a prompt and return a conversational, in-context response. They cover the generative workloads you would expect: summarization, drafting, extraction, classification-by-instruction, question answering, and tool or function calling. Oracle's chat catalogue spans several model families — Cohere, Meta (Llama), Google, OpenAI, and xAI — so that teams can trade off quality, latency, and cost without leaving the service.

Two Cohere families are worth knowing by name because they define the enterprise RAG pattern on OCI:

• Cohere Command: The general-purpose text generation family, built for business tasks including document summarization, copywriting, information extraction, content drafting, and conversational task fulfillment.
• Cohere Command R and Command R+: Purpose-built for enterprise Retrieval-Augmented Generation (RAG) and multi-step tool use (function calling). Their 128k (128,000) token context window lets a single prompt carry extensive technical documentation, legal agreements, or corporate manuals, and the models emit citations back to the supplied source passages, which is what makes grounded answers auditable.

2. Embedding Models

Embedding models convert text into dense numerical vectors so that meaning can be compared mathematically. They do not generate text.

• Cohere Embed: Translates unstructured text into dense vector representations, supporting over 100 languages and typically producing 1024-dimension vectors. These vectors power semantic search, clustering, deduplication, and the retrieval half of a RAG pipeline — including storage and search inside Oracle Database 23ai AI Vector Search (covered in section 6.3).

3. Rerank Models

Rerank models take a query plus a candidate list of retrieved passages and reorder that list by true relevance. In a RAG pipeline, vector search casts a wide net cheaply and the reranker sharpens the top results before they are handed to the chat model as context. Recognising reranking as a distinct step — retrieval, then reranking, then generation — is a common exam discriminator.

---

Hosting Modalities: On-Demand Serving vs. Dedicated AI Clusters

To accommodate diverse enterprise workloads, cost models, and performance requirements, OCI Generative AI provides two distinct hosting paradigms: On-Demand Serving and Dedicated AI Clusters.

On-Demand Serving (Multi-Tenant)

On-Demand Serving provides immediate, serverless access to pre-trained foundation models hosted across shared OCI GPU infrastructure:

• Shared Infrastructure: Workloads run on a secure, multi-tenant pool of GPUs managed and scaled automatically by Oracle.
• Pay-Per-Token Billing: Pricing is calculated purely on consumption—specifically, the number of input tokens processed and output tokens generated. Organizations incur zero infrastructure idle costs.
• Zero Provisioning Latency: Models are available instantly; developers can execute REST API calls immediately without waiting for compute clusters to initialize.
• Best Suited For: Prototyping, proof-of-concept (PoC) exploration, development environments, and applications with sporadic, unpredictable, or low-volume request traffic.
• Constraints: On-Demand Serving cannot host custom fine-tuned models and does not provide guaranteed latency service-level agreements (SLAs).

Dedicated AI Clusters (Single-Tenant)

Dedicated AI Clusters allocate dedicated compute nodes equipped with enterprise-grade GPUs (such as NVIDIA A100 Tensor Core GPUs with 80GB VRAM or NVIDIA H100 GPUs) exclusively to a single customer tenancy:

• Single-Tenant Isolation: GPU nodes are assigned exclusively to the customer, completely eliminating "noisy neighbor" interference and ensuring consistent, deterministic performance.
• Predictable Hourly Pricing: Billing is based on a fixed hourly rate per allocated GPU unit, providing consistent, predictable expenditures regardless of how many tokens are processed.
• Guaranteed Throughput and Latency SLAs: Organizations running mission-critical applications receive guaranteed compute capacity and bounded response times.
• Mandatory for Customization: Hosting custom fine-tuned models strictly requires a dedicated serving cluster. In addition, training or fine-tuning foundation models requires a dedicated training cluster.
• Virtual Cloud Network (VCN) Private Endpoints: Dedicated AI Clusters can be exposed via private IP addresses within an OCI Virtual Cloud Network (VCN), restricting traffic to internal corporate networks and avoiding public internet exposure.
• Commitment Horizon: Typically requires a minimum commitment duration (e.g., one month or specific unit allocation) to reserve physical GPU hardware.

---

Fine-Tuning Capabilities: LoRA and T-Few

While pre-trained foundation models exhibit broad world knowledge, enterprise applications often require specialized terminology, corporate style compliance, or domain-specific classification accuracy. Rather than conducting computationally expensive full-parameter fine-tuning, OCI Generative AI utilizes Parameter-Efficient Fine-Tuning (PEFT) techniques.

Full Fine-Tuning:                 Update 100% of base model parameters (billions of weights)
                                  -> Prohibitive GPU compute, massive VRAM, risk of catastrophic forgetting

Parameter-Efficient (PEFT/LoRA):  Freeze 100% of base model parameters
                                  -> Inject and train small low-rank adapter matrices (< 1% parameters)
                                  -> Rapid training, modest GPU requirements, preserves base knowledge

1. Low-Rank Adaptation (LoRA)

Low-Rank Adaptation (LoRA) freezes the pre-trained foundation model weights entirely and introduces trainable rank-decomposition matrices into each layer of the Transformer architecture. For a weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA decomposes the weight update into two low-rank matrices: $\Delta W = B \times A$, where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and the rank $r \ll \min(d, k)$.

• Reduced Resource Consumption: Reduces the number of trainable parameters by up to 99%, drastically lowering GPU memory requirements during training.
• Mitigates Catastrophic Forgetting: Because the core weights remain frozen, the model retains its underlying language capabilities while mastering specialized domain patterns.

2. T-Few Fine-Tuning

T-Few (Few-Shot Parameter-Efficient Tuning) is an advanced parameter-efficient method that updates a tiny fraction of weights using the $(IA)^3$ (Infused Adapter by Inhibiting and Amplifying Inner Activations) architecture:

• Extreme Parameter Efficiency: Trains less than 0.1% of the model's total parameter count by selectively scaling intermediate activation vectors.
• Small Data Regimes: Operates effectively on modest custom training datasets containing only hundreds or thousands of high-quality labeled examples.
• Rapid Convergence: Completes fine-tuning runs in a fraction of the time required by alternative approaches.

Fine-Tuning Execution Workflow

1. The user prepares a training dataset formatted in JSON Lines (.jsonl) containing prompt-completion pairs and uploads it to OCI Object Storage.
2. The administrator provisions a Dedicated AI Cluster with the Training cluster type.
3. The user initiates a fine-tuning job specifying the base model (e.g., Cohere Command), training data location, and hyperparameters.
4. Upon completion, the custom model weights are saved into the customer's compartment. The model can then be deployed to a Dedicated AI Cluster with the Hosting cluster type for production serving.

---

OCI Generative AI Playground

The OCI Generative AI Playground is an interactive web-based console integrated into the OCI Console. It enables developers, data scientists, and business analysts to experiment with foundation models, test prompt designs, and calibrate decoding parameters before writing application code.

Three Playground Modes

1. Generation Mode: Designed for free-form text completion, summarization, extraction, and drafting. Users provide a prompt, and the model generates a coherent continuation.
2. Chat Mode: Designed for conversational interactions. Provides a multi-turn chat interface supporting system instructions (defining persona, tone, and guardrails) and automatic conversation history management.
3. Embedding Mode: Takes raw text strings as input and displays the resulting dense numerical vector representations, allowing developers to inspect vector dimensionality and verify semantic clustering.

Key Decoding Hyperparameters

When evaluating models in the Playground or invoking them via REST APIs, practitioners calibrate several critical hyperparameters:

---

Model Portfolio & Hosting Modalities Comparison