5.3 OKE Workload Identity, Secrets & Persistent Storage

4.6 OKE Workload Identity, Secrets & Persistent Storage

Zero-Trust Pod Security & OCI Workload Identity

Securing containerized enterprise workloads requires moving beyond host-level trust boundaries. Historically, applications running in Kubernetes obtained access to OCI resources through one of two mechanisms: embedding static OCI API signing keys inside Kubernetes Secrets, or assigning Instance Principals to the underlying worker node compute pool.

Both legacy approaches introduce severe security vulnerabilities in multi-tenant enterprise environments:

• Static API Keys: Long-lived private keys stored in Kubernetes secrets or config files risk source code leakage, unauthorized exfiltration, and operational failures during manual credential rotation.
• Instance Principals Blast Radius: An Instance Principal grants IAM permissions to the physical or virtual worker node. Consequently, every pod co-located on that node inherits identical administrative privileges. A compromised frontend pod can exploit the host's instance principal to access sensitive Object Storage buckets or databases intended solely for backend financial microservices.

To eliminate this structural vulnerability, OKE provides OCI Workload Identity.

---

How Workload Identity Works

In OCI, a workload running on an OKE cluster is treated as a resource in its own right. Its identity is the unique combination of cluster, namespace, and Kubernetes service account. You use that combination in IAM policies:

1. Prepare the Workload

Create a namespace and a Kubernetes service account, and set the deployment so every pod of the workload uses that service account. Workload identity is only available in enhanced clusters.

2. Use a Supported OCI SDK

The application authenticates with the OKE workload identity provider in an OCI SDK. Supported SDKs include Go, Java, Python, .NET, and Ruby at recent versions. Workload identity works through the SDKs; you do not use it from the Console. The provider exchanges the pod's Kubernetes service account token for a short-lived OCI token and caches it, so Oracle recommends creating one provider instance and reusing it throughout the application.

3. Write a Workload-Scoped IAM Policy

The policy uses any-user with conditions that pin it to one workload:

Allow any-user to manage objects in compartment Financial-Data where all {
  request.principal.type = 'workload',
  request.principal.namespace = 'payments',
  request.principal.service_account = 'payment-processor-sa',
  request.principal.cluster_id = '<cluster-ocid>'
}

Including request.principal.cluster_id matters: without it, a workload with the same namespace and service account name in another cluster could match the policy. OCI Audit records the requests made by each workload identity, which supports compliance reporting.

Under this model, even if a compromised reporting pod runs on the exact same worker node as the payment processor, it cannot assume the payment processor's IAM role. Workload Identity enforces absolute cryptographic isolation at the individual pod level.

---

Centralized Secret Management via Secrets Store CSI & OCI Vault

Storing database credentials, API tokens, and TLS certificates directly in native Kubernetes Secret objects introduces compliance risks, as secrets are stored in etcd (often unencrypted or with basic base64 encoding unless envelope encryption is configured) and lack enterprise audit logging.

OKE integrates with OCI Vault through the Kubernetes Secrets Store CSI driver (secrets-store.csi.k8s.io) and Oracle's open-source OCI Secrets Store CSI Driver Provider. A SecretProviderClass lists the Vault secrets to fetch:

• Volume Mounts: Secrets are retrieved from Vault when the pod starts and mounted into the container as files in a CSI volume, so they do not need to be stored as Kubernetes Secret objects in etcd.
• Rotation Sync: The driver can periodically refresh mounted contents (and optionally synced Kubernetes secrets) from Vault at a configured frequency, so a secret rotated in Vault reaches running pods. This auto-rotation feature is off by default and is enabled with a Helm setting.
• Encryption: Vault secrets are encrypted with a master encryption key protected by FIPS 140-2 Level 3 hardware security modules, and access is controlled with IAM policies on the secret-family resources.

---

Persistent Storage in OKE: Container Storage Interface (CSI) Plugins

Stateful microservices require resilient, persistent storage that survives pod recreation and worker node failure. OKE provides production-ready implementations of the Container Storage Interface (CSI) specification, supporting both block-level and file-level storage.

1. OCI Block Volume CSI Driver (ReadWriteOnce - RWO)

The OCI Block Volume CSI driver provisions high-performance, low-latency persistent block storage backed by OCI Block Volumes:

• Access Mode: Operates exclusively under ReadWriteOnce (RWO). A block volume can be formatted with a filesystem (such as ext4 or xfs) and attached to a single worker node at any given time.
• Dynamic Provisioning: StorageClasses (oci-bv) automate volume provisioning. When a developer submits a PersistentVolumeClaim (PVC), the CSI plugin calls the OCI Block Volume API to provision the disk and attach it via iSCSI or paravirtualized attachment to the host node.
• Dynamic Performance Tiers: Architects configure storage throughput and IOPS dynamically through the vpusPerGB StorageClass parameter: 0 VPUs/GB (Low Cost) for batch logging; 10 VPUs/GB (Balanced) for general workloads; 20 VPUs/GB (High Performance) for relational databases; and 30 to 120 VPUs/GB (Ultra High Performance) delivering up to 300,000 IOPS per volume for mission-critical transactional engines.
• Online Volume Expansion: Persistent volumes can be resized dynamically by editing the PVC storage request without taking the pod offline.

2. OCI File Storage Service (FSS) CSI Driver (ReadWriteMany - RWX)

The OCI File Storage Service CSI driver provides managed, highly available network file systems conforming to the POSIX specification:

• Access Mode: Supports ReadWriteMany (RWX). Multiple pods distributed across different worker nodes, different Fault Domains, and even different Availability Domains can simultaneously mount and perform concurrent reads and writes against the same shared filesystem.
• Dynamic Provisioning: The oci-fss StorageClass dynamically provisions FSS file systems, export paths, and mount targets within the customer's VCN.
• Mount targets: A File Storage mount target lives in a subnet and serves NFS exports. Pods on nodes in other subnets or availability domains can mount it as long as routing and security rules allow NFS traffic. For DR, File Storage replication copies a file system to another region.
• Ideal Workloads: Content management systems (WordPress, Drupal), distributed machine learning datasets, shared artifact repositories, and microservices requiring concurrent write access to shared directories.

---

Volume Snapshots, Cloning & Backup Automation

OKE storage architecture integrates natively with OCI backup automation through Kubernetes Custom Resource Definitions (CRDs):

• CSI Volume Snapshots: Developers define declarative VolumeSnapshot objects in Kubernetes. The OCI Block Volume CSI plugin captures a point-in-time, crash-consistent snapshot of the volume and stores it redundantly in OCI Object Storage.
• Volume Cloning: A new PVC can specify an existing VolumeSnapshot or another PVC as its dataSource, and the CSI driver creates a new block volume from that source, which is useful for creating test copies of production data.

---

Storage & Security Architectural Tradeoff Matrix