2.6 Custom Resource Definitions (CRDs) & Operators

2.6 Custom Resource Definitions (CRDs) & Operators

Kubernetes can add new resource types without modifying the core API server binary. A CustomResourceDefinition (CRD) registers the API path, names, scope, versions, and validation schema for a new type. A custom resource (CR) is one object of that type. An Operator combines custom resources with a controller that encodes operational knowledge in a reconciliation loop.

1. CRD, Custom Resource, and Controller

Creating a CRD gives the new type standard Kubernetes API behavior: discovery, CRUD operations, watch, authentication, RBAC authorization, admission, labels, annotations, and finalizers. A CRD alone stores structured desired state; it does not create Deployments, databases, or external infrastructure unless a controller watches it.

2. Reading a CRD Manifest

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.platform.example.com
spec:
  group: platform.example.com
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    kind: Backup
    shortNames:
    - bkp
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required:
            - targetPVC
            properties:
              targetPVC:
                type: string
              retentionDays:
                type: integer
                minimum: 1
    subresources:
      status: {}

The CRD name must be <plural>.<group>. Namespaced resources live in one namespace; Cluster resources do not. Every listed version says whether the API server serves it and whether it is the storage version. Exactly one version must have storage: true. Multiple served versions may require a conversion strategy so clients see a coherent schema while stored objects migrate.

The structural OpenAPI schema rejects invalid types and can require fields, constrain numbers, or define nested objects. The status subresource lets the controller report observed state separately from the user's desired spec.

3. Create and Discover Custom Resources

After the CRD is Established, a custom object can be created like any built-in resource:

apiVersion: platform.example.com/v1
kind: Backup
metadata:
  name: nightly-orders
  namespace: data
spec:
  targetPVC: orders-data
  retentionDays: 14

Useful discovery commands:

kubectl get crd backups.platform.example.com
kubectl wait --for=condition=Established crd/backups.platform.example.com
kubectl api-resources --api-group=platform.example.com
kubectl explain backup.spec
kubectl get backups -A
kubectl get backup nightly-orders -n data -o yaml

If the API server returns “no matches for kind,” verify that the CRD exists, the apiVersion group/version is served, discovery has refreshed, and the Kind spelling matches.

4. Operator Reconciliation and Finalizers

A controller watches desired state and acts until observed state matches it:

Backup CR created
  -> controller observes generation
  -> validates referenced PVC
  -> creates Job / snapshot request
  -> records status.conditions and observedGeneration
  -> requeues on changes or failures

Controllers must be idempotent: running reconciliation again should converge rather than duplicate work. They commonly use owner references so Kubernetes garbage collection removes dependent objects with their owner. A finalizer delays deletion while a controller performs external cleanup. If a CR remains Terminating, inspect metadata.finalizers, controller health, RBAC, and logs before removing a finalizer manually; forced removal can orphan cloud resources or data.

5. Installation and Upgrade Order

A reliable installation order is:

1. Apply the CRD and wait for the Established condition.
2. Install the controller Deployment, ServiceAccount, and RBAC.
3. Confirm the controller is Available and review its logs.
4. Create custom resources only after their API is served.
5. Inspect each CR's status.conditions, Events, generated children, and controller logs.

Helm charts can place CRDs in a top-level crds/ directory so they are installed before templated resources. Helm deliberately treats CRD lifecycle cautiously; do not assume a normal chart upgrade or uninstall will safely migrate or remove CRDs and their data. Follow the component owner's upgrade path.

For multi-version upgrades, add the new served version and conversion support, migrate stored objects, update the storage version, verify status.storedVersions, and only then stop serving an old version. Deleting a CRD deletes its custom-resource data from the API, so export or back up required objects first.

6. Operator Troubleshooting Runbook

kubectl get crd <name> -o yaml
kubectl get deployment -n <operator-namespace>
kubectl auth can-i --as=system:serviceaccount:<ns>:<sa> get <resource> -A
kubectl logs -n <operator-namespace> deployment/<controller>
kubectl describe <custom-resource> <name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Separate failure layers: the CRD may not be Established, the controller may be unavailable, RBAC may deny watches or writes, admission may reject generated objects, a finalizer may block deletion, or the custom resource may reference a missing dependency. Conditions and controller logs usually reveal which layer failed.