Learn how to - How To Upgrade OpenShift / OKD Cluster Minor Version

Felix Williams Containers Kubernetes openshift April 28, 2023

Learn how to – How To Upgrade OpenShift / OKD Cluster Minor Version.

After a successful installation and configuration of OpenShift Container Platform, the updates are providedover-the-air by OpenShift Update Service (OSUS). The operator responsible for checking valid updates available for your cluster with the OpenShift Update Service is called Cluster Version Operator (CVO). When you request an update, the CVO uses the release image for that update to upgrade your cluster. All the release artifacts are stored as container images in the Quay registry.

It is important to note that the OpenShift Update Service displays all valid updates for your Cluster version. It is highly recommended that you do not force an update to a version that the OpenShift Update Service does not display. This is because a suitability check is performed to guarantee functional cluster after the upgrade. During the upgrade process, the Machine Config Operator (MCO) applies the new configuration to your cluster machines.

Before you start a minot upgrade to your OpenShift Cluster, check the current cluster version using oc command line tool if configured or from a web console. You should have the cluster admin rolebinding to use these functions.

We have the following OpenShift / OKD installation guides on our website:

1) Confirm current OpenShift Cluster version

Check the current version and ensure your cluster is available:

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.5     True        False         24d     Cluster version is 4.8.5

The current version of OpenShift Container Platform installed can also be checked from the web console – Administration → Cluster Settings > Details

How To Upgrade OpenShift / OKD Cluster Minor Version 1

Also check available Cluster nodes and their current status. Ensure they are all in Ready State before you can initiate an upgrade.

$ oc get nodes
NAME                           STATUS   ROLES    AGE   VERSION
master01.ocp4.techwizpro.com   Ready    master   24d   v1.21.1+9807387
master02.ocp4.techwizpro.com   Ready    master   24d   v1.21.1+9807387
master03.ocp4.techwizpro.com   Ready    master   24d   v1.21.1+9807387
worker01.ocp4.techwizpro.com   Ready    worker   24d   v1.21.1+9807387
worker02.ocp4.techwizpro.com   Ready    worker   24d   v1.21.1+9807387
worker03.ocp4.techwizpro.com   Ready    worker   24d   v1.21.1+9807387

2) Backup Etcd database data

Access one of the control plane nodes(master node) using oc debug command to start a debug session:

$ oc debug node/<node_name>

Here is an example with expected output:

$ oc debug node/master01.ocp4.example.com
Starting pod/master01ocp4examplecom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.100.11
If you don't see a command prompt, try pressing enter.
sh-4.4#

Change your root directory to the host:

sh-4.4# chroot /host

Then initiate backup of etcd data using provided script namedcluster-backup.sh:

sh-4.4# which cluster-backup.sh
/usr/local/bin/cluster-backup.sh

The cluster-backup.sh script is part of etcd Cluster Operator and it is just a wrapper around the etcdctl snapshot save command. Execute the script while passing the backups directory:

sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

Here is the output as captured from my backup process

found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-19
found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-9
found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
3f8cc62fb9dd794113201bfabd8af4be0fdaa523987051cdb358438ad4e8aca6
etcdctl version: 3.4.14
API version: 3.4
{"level":"info","ts":1631392412.4503953,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-09-11_203329.db.part"}
{"level":"info","ts":"2021-09-11T20:33:32.461Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1631392412.4615548,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://157.90.142.231:2379"}
{"level":"info","ts":"2021-09-11T20:33:33.712Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1631392413.9274824,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://157.90.142.231:2379","size":"102 MB","took":1.477013816}
{"level":"info","ts":1631392413.9344463,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-09-11_203329.db"}
Snapshot saved at /home/core/assets/backup/snapshot_2021-09-11_203329.db
{"hash":3708394880,"revision":12317584,"totalKey":7946,"totalSize":102191104}
snapshot db and kube resources are successfully saved to /home/core/assets/backup

Check if the backup files are available in our backups directory:

sh-4.4# ls -lh /home/core/assets/backup/
total 98M
-rw-------. 1 root root 98M Sep 11 20:33 snapshot_2021-09-11_203329.db
-rw-------. 1 root root 92K Sep 11 20:33 static_kuberesources_2021-09-11_203329.tar.gz

The files as seen are:

snapshot_<datetimestamp>.db: The etcd snapshot file.
static_kuberesources_<datetimestamp>.tar.gz: File that contains the resources for the static pods. When etcd encryption is enabled, the encryption keys for the etcd snapshot will be contained in this file.

You can copy the backup files to a separate system or location outside the server for better security if the node becomes unavailable during upgrade.

3) Changing Updates Channel (Optional)

The OpenShift Container Platform offers the following upgrade channels:

candidate
fast
stable

Review the current update channel information and confirm that your channel is set to stable-4.8:

$ oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "fast-4.8",
  "clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585"
}

You can decide to change an upgrade channel before the actual upgrade of the cluster.

From Command Line Interface

Switch Update channel from CLI using patch:

oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "<channel>”}]'

# Example
$ oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "stable-4.8"}]'
clusterversion.config.openshift.io/version patched

$ oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "stable-4.8",
  "clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585"
}

From Web Console

How To Upgrade OpenShift / OKD Cluster Minor Version 2

NOTE:For production clusters, you must subscribe to a stable-* or fast-* channel. Your cluster is fully supported by Red Hat subscription if you change from stable to fast channel.

In my example below I’ve set the channel to fast-4.8.

How To Upgrade OpenShift / OKD Cluster Minor Version 3

4) Perform Minor Upgrade on OpenShift / OKD Cluster

You can choose to perform a cluster upgrade from:

Bastion Server / Workstation oc command line
From OpenShift web console

Upgrade your OpenShift Container Platform from CLI

Check available upgrades

$ oc adm upgrade
Cluster version is 4.8.5

Updates:

VERSION IMAGE
4.8.9   quay.io/openshift-release-dev/ocp-release@sha256:5fb4b4225498912357294785b96cde6b185eaed20bbf7a4d008c462134a4edfd
4.8.10  quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db

As seen we have two minor upgrades that can be performed:

To version 4.8.9
To version 4.8.10

The easiest way to upgrade is to the latest version:

$ oc adm upgrade --to-latest=true 
Updating to latest version 4.8.10

To update to a specific version:

$ oc adm upgrade --to=<version> 
#e.g 4.8.9, I'll run:
$ oc adm upgrade --to=4.8.9

You can easily review Cluster Version Operator status with the following command:

$ oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "stable-4.8",
  "clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585",
  "desiredUpdate": {
    "force": false,
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db",
    "version": "4.8.10"
  }
}

The oc adm upgrade command will give progress update with the steps:

$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.8.10: 69 of 678 done (10% complete)
Updates:

VERSION IMAGE
4.8.9   quay.io/openshift-release-dev/ocp-release@sha256:5fb4b4225498912357294785b96cde6b185eaed20bbf7a4d008c462134a4edfd
4.8.10  quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db

Upgrade OpenShift Container Platform from UI

Administration → Cluster Settings→ Details→ Select channel→ Select a version to update to, and click Save. The Input channel Update status changes to Update to <product-version> in progress.

How To Upgrade OpenShift / OKD Cluster Minor Version 4

All cluster operators will be upgraded one after the other until all are in the minor version selected during upgrade:

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.5     True        False         False      119m
baremetal                                  4.8.5     True        False         False      24d
cloud-credential                           4.8.5     True        False         False      24d
cluster-autoscaler                         4.8.5     True        False         False      24d
config-operator                            4.8.5     True        False         False      24d
console                                    4.8.5     True        False         False      36h
csi-snapshot-controller                    4.8.5     True        False         False      24d
dns                                        4.8.5     True        False         False      24d
etcd                                       4.8.10    True        False         False      24d
image-registry                             4.8.5     True        False         False      24d
ingress                                    4.8.5     True        False         False      24d
insights                                   4.8.5     True        False         False      24d
kube-apiserver                             4.8.5     True        False         False      24d
kube-controller-manager                    4.8.5     True        False         False      24d
kube-scheduler                             4.8.5     True        False         False      24d
kube-storage-version-migrator              4.8.5     True        False         False      4d16h
machine-api                                4.8.5     True        False         False      24d
machine-approver                           4.8.5     True        False         False      24d
machine-config                             4.8.5     True        False         False      24d
marketplace                                4.8.5     True        False         False      24d
monitoring                                 4.8.5     True        False         False      <invalid>
network                                    4.8.5     True        False         False      24d
node-tuning                                4.8.5     True        False         False      24d
openshift-apiserver                        4.8.5     True        False         False      32h
openshift-controller-manager               4.8.5     True        False         False      23d
openshift-samples                          4.8.5     True        False         False      24d
operator-lifecycle-manager                 4.8.5     True        False         False      24d
operator-lifecycle-manager-catalog         4.8.5     True        False         False      24d
operator-lifecycle-manager-packageserver   4.8.5     True        False         False      7d11h
service-ca                                 4.8.5     True        False         False      24d
storage                                    4.8.5     True        False         False      24d

5) Validate OpenShift CLuster Upgrade

Wait for the upgrade process to complete then confirm that the cluster version has updated to the new version:

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.10    True        False         37h     Cluster version is 4.8.10

Checking cluster version from the web console

How To Upgrade OpenShift / OKD Cluster Minor Version 5

To obtain more detailed information about the cluster status run the command:

$ oc describe clusterversion

If you try running the command oc adm upgrade immediately after upgrade to the latest release you should get a message similar to below:

$ oc adm upgrade
Cluster version is 4.8.10

No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.

Conclusion

In this short guide we’ve shown how one can easily perform minor upgrade of OpenShift container cluster version. The process can be initiated from a web console or from the command line, it all depends on your preference. In our articles to follow we’ll cover steps required to perform Major versions upgrade in anOpenShift container cluster.

More guides on OpenShift / OKD container Platform:

Cart

Learn how to – How To Upgrade OpenShift / OKD Cluster Minor Version