2.1 Kubernetes TLS Certificates & PKI Infrastructure

Kubernetes TLS Certificates & PKI Infrastructure

Security in Kubernetes begins at the transport layer. Every control plane interaction—from worker node kubelet heartbeats to administrator kubectl commands and inter-component control loops—is encrypted and mutually authenticated using Transport Layer Security (TLS).

A deep understanding of the cluster's Public Key Infrastructure (PKI) layout, certificate authority boundaries, certificate expiration cycles, OpenSSL diagnostic inspection commands, and the native CertificateSigningRequest (CSR) API is fundamental for passing the Certified Kubernetes Administrator (CKA) exam and running reliable production clusters.

---

1. Kubernetes PKI Architecture & Dedicated CA Boundaries

A standard cluster initialized with kubeadm provisions its cryptographic assets in /etc/kubernetes/pki. Rather than utilizing a single universal Certificate Authority (CA) for all operations, Kubernetes enforces distinct cryptographic trust domains to maintain isolation between distinct subsystems.

+-----------------------------------------------------------------------------------------+
|                        KUBERNETES PKI ARCHITECTURE & CA BOUNDARIES                      |
|                                                                                         |
|   +------------------------------------+   +----------------------------------------+   |
|   |         MAIN CLUSTER CA            |   |                ETCD CA                 |   |
|   |   /etc/kubernetes/pki/ca.crt       |   |     /etc/kubernetes/pki/etcd/ca.crt    |   |
|   |   /etc/kubernetes/pki/ca.key       |   |     /etc/kubernetes/pki/etcd/ca.key    |   |
|   +-----------------+------------------+   +-------------------+--------------------+   |
|                     |                                          |                        |
|       +-------------+-------------+              +-------------+-------------+          |
|       |                           |              |                           |          |
|       v                           v              v                           v          |
|  apiserver.crt            apiserver-kubelet-  server.crt (etcd)          peer.crt (etcd)|
|  (API Server Server)      client.crt          (etcd Server)              (Peer-to-Peer) |
|                           (API -> Kubelet)                                              |
|                                                  +----------------------------------+   |
|   +------------------------------------+         |       SERVICEACCOUNT KEYPAIR     |   |
|   |           FRONT-PROXY CA           |         |   /etc/kubernetes/pki/sa.key     |   |
|   | /etc/kubernetes/pki/               |         |   /etc/kubernetes/pki/sa.pub     |   |
|   |   front-proxy-ca.crt / .key        |         | (Signs/verifies ServiceAccount   |   |
|   +-----------------+------------------+         |  projected JWT tokens)           |   |
|                     |                            +----------------------------------+   |
|                     v                                                                   |
|        front-proxy-client.crt                                                           |
|        (API Aggregator Proxy)                                                           |
+-----------------------------------------------------------------------------------------+

The Four Cryptographic Trust Domains:

1. Main Root Cluster CA (/etc/kubernetes/pki/ca.crt & ca.key):
   • Signs the API server serving certificate (apiserver.crt).
   • Signs the API server's client certificate used to communicate with kubelets (apiserver-kubelet-client.crt).
   • Signs client certificates for cluster administrators and internal controllers (admin.conf, controller-manager.conf, scheduler.conf).
   • Signs client certificates submitted by worker node kubelets during node bootstrapping.
2. etcd CA (/etc/kubernetes/pki/etcd/ca.crt & etcd/ca.key):
   • Signs the etcd server certificate (etcd/server.crt).
   • Signs peer certificates used for mutual TLS communication between etcd cluster members (etcd/peer.crt).
   • Signs the client certificate used by kube-apiserver to read and write state to etcd (apiserver-etcd-client.crt).
3. Front-Proxy CA (/etc/kubernetes/pki/front-proxy-ca.crt & front-proxy-ca.key):
   • Used exclusively by the Kubernetes API Aggregator proxy layer.
   • Authenticates the API server when proxying incoming extension requests to aggregated API servers (such as metrics-server or custom CRD controllers).
4. ServiceAccount Signing Keypair (/etc/kubernetes/pki/sa.key & sa.pub):
   • This is not a certificate authority, but a 2048-bit RSA keypair (or ECDSA keypair).
   • sa.key is used by the kube-controller-manager (Token Controller) to cryptographically sign JSON Web Tokens (JWTs) issued to ServiceAccounts.
   • sa.pub is loaded by the kube-apiserver to validate JWT signatures presented by pods in HTTP Authorization: Bearer <token> headers.

---

2. Complete Inventory of PKI Assets in /etc/kubernetes/pki

The following table outlines every certificate and key provisioned on a standard Kubernetes control plane node:

[!IMPORTANT] Subject Alternative Names (SANs) Requirement: The API server certificate (apiserver.crt) must include all reachable IP addresses and DNS names under its X509v3 Subject Alternative Name extension. By default, kubeadm injects:

• DNS Names: kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, and the control plane hostname.
• IP Addresses: The first IP address of the Service CIDR (e.g., 10.96.0.1), the control plane node's primary IPv4 address, and 127.0.0.1.

If an administrator connects to the API server via an external load balancer IP or custom domain name that is missing from the SAN list, the TLS handshake fails immediately with the error: x509: certificate is valid for X, not Y.

---

3. Certificate Expiration & Lifecycle Management with Kubeadm

All control plane and client certificates generated by kubeadm during initial bootstrapping have a validity lifespan of 1 year (365 days). The root CA certificates (ca.crt, etcd/ca.crt, front-proxy-ca.crt) are valid for 10 years.

Inspecting Certificate Expiration Status

To view the exact expiration timestamp, remaining residual time, and backing CA for all cluster certificates, execute:

sudo kubeadm certs check-expiration

Example command output:

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 15, 2027 08:30 UTC   364d            ca                      no
apiserver                  Jan 15, 2027 08:30 UTC   364d            ca                      no
apiserver-etcd-client      Jan 15, 2027 08:30 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 15, 2027 08:30 UTC   364d            ca                      no
controller-manager.conf    Jan 15, 2027 08:30 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 15, 2027 08:30 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 15, 2027 08:30 UTC   364d            etcd-ca                 no
etcd-server                Jan 15, 2027 08:30 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 15, 2027 08:30 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 15, 2027 08:30 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 12, 2036 08:30 UTC   9y              no
etcd-ca                 Jan 12, 2036 08:30 UTC   9y              no
front-proxy-ca          Jan 12, 2036 08:30 UTC   9y              no

Renewing Control Plane Certificates

kubeadm allows automated renewal of all certificates or individual targeted certificates:

# 1. Renew ALL control plane certificates and kubeconfig files simultaneously
sudo kubeadm certs renew all

# Alternatively, renew only a specific component certificate
sudo kubeadm certs renew apiserver
sudo kubeadm certs renew apiserver-kubelet-client

# 2. Update the local administrator kubeconfig with the renewed credentials
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

[!CAUTION] Restarting Static Pods is Mandatory After Renewal: Renewing certificate files on disk does not automatically update running processes. Control plane binaries (kube-apiserver, kube-controller-manager, kube-scheduler, etcd) load their certificates into memory at startup. After running kubeadm certs renew all, you must restart the control plane static pods.

The safest method in production is to temporarily move the static pod manifests out of /etc/kubernetes/manifests/ to a backup location, wait 10 seconds for the kubelet to terminate the containers, and then move them back. Restarting kubelet alone does not guarantee that unchanged static-Pod containers reload certificates. Use the manifest move-out/move-back procedure and verify each recreated container.

---

4. Low-Level Certificate Diagnostics with OpenSSL

In troubleshooting scenarios or during the CKA exam, you may need to inspect raw certificate properties without kubeadm. The OpenSSL CLI provides direct visibility into certificate fields:

# 1. View entire certificate details in readable text
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

# 2. Check certificate validity dates (Not Before, Not After)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates

# 3. Inspect Subject (Common Name and Organization)
openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -subject
# Output: subject=CN = kube-apiserver-kubelet-client, O = system:masters

# 4. Check the signing Issuer CA
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -issuer
# Output: issuer=CN = kubernetes

# 5. Extract Subject Alternative Names (SANs)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName
# Output: DNS:master-node, DNS:kubernetes, DNS:kubernetes.default, IP Address:10.96.0.1, IP Address:192.168.1.10

---

5. Kubernetes Certificates API & CSR Lifecycle (certificates.k8s.io/v1)

Granting direct access to /etc/kubernetes/pki/ca.key to sign certificates for developers or applications violates the principle of least privilege. The Kubernetes Certificates API provides a declarative mechanism where clients submit a CertificateSigningRequest (CSR) object, and administrators review and approve it using kubectl.

+-----------------------------------------------------------------------------------------+
|                     CERTIFICATESIGNINGREQUEST (CSR) LIFECYCLE FLOW                      |
|                                                                                         |
|  [1. CLIENT / OPERATOR]                                                                 |
|  - Generate RSA Private Key: openssl genrsa -out john.key 2048                          |
|  - Generate CSR: openssl req -new -key john.key -out john.csr -subj "/CN=john/O=devs"   |
|  - Base64 encode CSR: cat john.csr | base64 | tr -d '\n'                                |
|                            |                                                            |
|                            v                                                            |
|  [2. SUBMIT K8S MANIFEST] (apiVersion: certificates.k8s.io/v1)                          |
|  - spec.request: <BASE64_STRING>                                                        |
|  - spec.signerName: kubernetes.io/kube-apiserver-client                                  |
|  - spec.usages: ["client auth"]                                                         |
|                            |                                                            |
|                            v                                                            |
|  [3. KUBE-APISERVER]    ---> CSR stored in etcd (Status: Pending)                       |
|                            |                                                            |
|                            v                                                            |
|  [4. CLUSTER ADMIN]     ---> kubectl certificate approve john-csr                       |
|                            |                                                            |
|                            v                                                            |
|  [5. SIGNING CONTROLLER]---> Signs certificate using cluster ca.key                     |
|                            - Populates status.certificate field                         |
|                            |                                                            |
|                            v                                                            |
|  [6. CLIENT RETRIEVAL]  ---> kubectl get csr john-csr -o jsonpath='{.status.certificate}'|
|                            | base64 --decode > john.crt                                 |
+-----------------------------------------------------------------------------------------+

Step-by-Step CSR Workflow:

Step 1: Generate Client Private Key and OpenSSL CSR

# Generate 2048-bit RSA private key
openssl genrsa -out john.key 2048

# Generate CSR specifying user name (CN) and group membership (O)
openssl req -new -key john.key -out john.csr -subj "/CN=john/O=developers/O=qa"

Step 2: Base64 Encode the CSR

# Strip newlines to avoid YAML parsing errors
CSR_BASE64=$(cat john.csr | base64 | tr -d '\n')

Step 3: Create and Apply the CSR Resource

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: john-developer-csr
spec:
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS... # Embedded $CSR_BASE64 string
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400 # 24 hours (optional)
  usages:
  - client auth

kubectl apply -f john-developer-csr.yaml

Step 4: Inspect, Approve, or Deny the CSR

# List all pending CSRs
kubectl get csr

# Approve the request
kubectl certificate approve john-developer-csr

# If a request is unauthorized, deny it:
# kubectl certificate deny john-developer-csr

Step 5: Extract and Decode the Signed Certificate

# Query the status.certificate field and pipe to base64 decode
kubectl get csr john-developer-csr -o jsonpath='{.status.certificate}' | base64 --decode > john.crt

# Verify the certificate properties
openssl x509 -in john.crt -text -noout | grep -E "Subject:|Issuer:"