1.6 Cluster Lifecycle & Step-by-Step Upgrades

1.6 Cluster Lifecycle & Step-by-Step Upgrades

A Kubernetes upgrade changes both cluster-wide control-plane state and node-local binaries. Treat it as a sequence of small, verified changes. For the CKA environment, first inspect the installed versions and the package versions actually available on the host; do not blindly paste a package revision from memory.

1. Supported Version Path and Skew

Kubernetes uses MAJOR.MINOR.PATCH versions. A kubeadm-managed cluster upgrades one minor version at a time. For example, upgrade v1.34.x to v1.35.x. If a cluster were two minors behind, complete and verify the intermediate upgrade before starting the next one. A skipped-minor path is unsupported; do not claim that it guarantees a particular failure such as etcd corruption.

During a rolling upgrade, temporary version skew is expected:

• kube-apiserver is the reference version and must not be older than another API server instance.
• kube-controller-manager and kube-scheduler may be one minor older than the API server, but should not be newer.
• kubelet may be up to three minor releases older than the API server and must not be newer.
• kubectl is supported within one minor release of the API server.

Always confirm the policy for the source and target releases before a production change.

2. Order of Operations

Use this order:

1. Upgrade kubeadm on the first control-plane node.
2. Run kubeadm upgrade plan, then kubeadm upgrade apply <target> once on that first node.
3. Drain that node when it runs ordinary workloads, upgrade kubelet and kubectl, restart kubelet, verify, and uncordon.
4. Repeat on each additional control-plane node, but run kubeadm upgrade node rather than upgrade apply.
5. Upgrade workers one at a time: drain, upgrade kubeadm, run kubeadm upgrade node, upgrade kubelet and kubectl, restart, verify, and uncordon.

The first control-plane node updates cluster-wide configuration and component manifests. The node command updates local configuration; it does not repeat the cluster-wide apply phase.

3. Discover the Exact Package Revision

On an APT-based node, use the repository output rather than guessing the suffix:

sudo apt-get update
apt-cache madison kubeadm | head
kubeadm version
kubectl get nodes

For a v1.35 target, select the exact v1.35.x package string shown by apt-cache madison. The variable below represents that exact value:

TARGET_PKG='1.35.x-*'       # replace with the exact listed package version
TARGET_K8S='v1.35.x'        # replace x with the selected patch
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm="$TARGET_PKG"
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply "$TARGET_K8S"

Do not run upgrade apply until the plan shows the intended target and required preflight checks are understood.

4. Upgrade the First Node's Kubelet

Check disruption policy before draining:

kubectl get pdb -A
kubectl drain controlplane --ignore-daemonsets

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet="$TARGET_PKG" kubectl="$TARGET_PKG"
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl get nodes -o wide
kubectl get pods -A
kubectl uncordon controlplane

A drain stops at pods using emptyDir unless --delete-emptydir-data acknowledges that loss. It may also stop at unmanaged pods unless --force is supplied. Those flags are not inherently “safe”; inspect the pods and use them only when the task permits the consequences.

5. Additional Control-Plane Nodes and Workers

After installing the target kubeadm package on an additional node, run:

sudo kubeadm upgrade node

Then upgrade kubelet and kubectl using the same package procedure. For a worker, initiate the drain and final uncordon from a terminal with cluster access:

kubectl drain worker-1 --ignore-daemonsets
ssh worker-1
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm="$TARGET_PKG"
sudo apt-mark hold kubeadm
sudo kubeadm upgrade node
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet="$TARGET_PKG" kubectl="$TARGET_PKG"
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
exit
kubectl get node worker-1 -o wide
kubectl uncordon worker-1

6. Package and Repository Cautions

Package revisions differ between operating systems and repositories. A Debian package suffix is not a Kubernetes semantic version, and the installed kubeadm, kubelet, and kubectl packages should resolve to the same intended minor and patch. On RPM-based hosts, translate the package steps to the configured package manager rather than using APT syntax. Preserve the repository configuration supplied by the task.

6. Verification and Failure Boundaries

After every node, confirm that it becomes Ready, its reported kubelet version is correct, control-plane and add-on pods are healthy, and workloads regain desired replicas. If a step fails, stop the rollout and diagnose that node. Package downgrades do not reverse API migrations; recovery should follow the target release's documented rollback or restore procedure, backed by an etcd snapshot taken before the upgrade.