5.3 Extending Kubernetes: CRDs, Operators & the Controller Pattern

5.3 Extending Kubernetes: CRDs, Operators & the Controller Pattern

Quick Answer: A CustomResourceDefinition (CRD) teaches kube-apiserver a new object type — after applying one you can kubectl get postgresclusters exactly as you would kubectl get pods. A CRD alone is only storage; a controller watching that resource is what gives it behaviour. Package the two together with real operational expertise and you have an Operator.

This is why the ecosystem looks the way it does. Prometheus, Argo CD, Istio, cert-manager, KEDA, Crossplane and Cluster API are all just CRDs plus controllers. Understanding the pattern once explains the whole CNCF landscape.

---

1. The Reconciliation Loop Is the Whole Idea

Every controller in Kubernetes — built-in or third-party — runs the same loop:

  ┌──────────────────────────────────────────────────────┐
  │  1. OBSERVE   watch the API server for the desired   │
  │               state (the object's .spec)             │
  │  2. COMPARE   read the actual state of the world     │
  │  3. ACT       take action to close the gap           │
  │  4. REPORT    write what happened into .status       │
  └──────────────────────┬───────────────────────────────┘
                         └──────────► repeat forever

The ReplicaSet controller compares desired replicas with running Pods and creates or deletes some. A backup Operator compares "a backup should exist every 6 hours" with the backups it can see and runs pg_dump. Same loop, different domain knowledge. Because the loop is level-triggered rather than edge-triggered, a controller that misses an event still converges on the next pass — which is why Kubernetes is resilient to restarts.

---

2. CustomResourceDefinitions

A CRD is itself a Kubernetes object that registers a new API type.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.data.example.com
spec:
  group: data.example.com
  scope: Namespaced                 # or Cluster
  names:
    plural: backups
    singular: backup
    kind: Backup
    shortNames: [bk]
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: [schedule, target]
            properties:
              schedule: { type: string }
              target:   { type: string }
              retain:   { type: integer, minimum: 1, default: 7 }

Once this is applied, the new type is a first-class API citizen:

kubectl get backups -A
kubectl describe backup nightly-orders
kubectl explain backup.spec.retain

Everything that works on built-in objects now works here too — because it all flows through the same API server:

• RBAC — you can grant get/list on backups in the data.example.com API group.
• Validation — the OpenAPI v3 schema is enforced at admission; retain: "seven" is rejected.
• Labels, selectors, annotations, owner references, garbage collection — all apply.
• kubectl, Helm, Kustomize, Argo CD, audit logging — all work unchanged.

The point candidates miss: applying that CRD creates nothing. You can create Backup objects and they will sit in etcd doing absolutely nothing until a controller watches them. A CRD is a schema; the controller is the behaviour.

---

3. The Operator Pattern

An Operator is a controller that encodes the knowledge a skilled human operator would apply to a specific application. The canonical description: an Operator is a way of packaging, deploying, and managing a Kubernetes application, automating the tasks a human expert performs.

A database Operator does not merely start Pods. It knows how to:

• provision a primary and its replicas with correct replication settings,
• promote a replica when the primary fails, and reconfigure the rest,
• take scheduled backups and verify them,
• perform a major-version upgrade in the right order with the right pre-flight checks,
• resize storage and reload configuration without data loss.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-db
spec:
  instances: 3
  storage:
    size: 100Gi
  backup:
    retentionPolicy: "30d"

Those ten lines replace a runbook. The Operator reconciles them into StatefulSets, Services, Secrets, PVCs, and PodDisruptionBudgets, then keeps watching.

Operator Maturity

The community describes Operator capability in five ascending levels: Basic Install → Seamless Upgrades → Full Lifecycle (backup, failover, restore) → Deep Insights (metrics, alerts, log processing) → Auto Pilot (auto-scaling, auto-tuning, auto-remediation). Most production Operators sit at level 3 or 4.

Building Operators

Kubebuilder and the Operator SDK scaffold Go-based Operators; KUDO, Metacontroller, and shell-operator support lower-code approaches; OperatorHub.io is the community catalogue. Operator Lifecycle Manager (OLM) installs and upgrades Operators themselves.

---

4. The Three Extension Surfaces

The practical distinction: use a CRD when you need a new kind of object; use a webhook when you need to change or police objects that already exist; use aggregation when the data is computed on demand and should not be persisted in etcd — which is exactly why live metrics use it.

---

5. Everyday CRDs You Should Recognise

When you meet an unfamiliar kubectl get somethings in a CNCF project, the answer is almost always "a CRD, reconciled by that project's controller".